Improving lifetimes performance on UCI dataset

[vierivincent15]

First of all, I would like to thank everyone on the team for creating such a useful library. It saves me a lot of time in building a CLV estimator project.

So, I was trying out a simpler model (BSM) to estimate CLV and comparing it to the current lifetimes model that I have to gain confidence that lifetimes is the go-to model to estimate CLV. However, things does not go as expected. Evaluation of both model with RMSE shows that BSM performs better than lifetimes model.

This is the dataset that I use for the evaluation: http://archive.ics.uci.edu/ml/datasets/online+retail

This is some background information about the Basic Structural Model: https://flora.insead.edu/fichiersti_wp/inseadwp2013/2013-27.pdf#page=20 (the element of cost is ignored in the evaluation, thefore C = 0)

This is how BSM looks in python code:

def bsm_clv(R, months):

  n = months  # months
  d = 0.01    # monthly discount rate

  # R is the average monthly purchase

  # Basic Structural Model Formula
  clv = R * np.sqrt(1+d) * (np.power(1+d, n) - 1) / (d * np.power(1+d, n))

  output = np.round(clv, 2)

  return output

I have attached a Notebook that can run the evaluation process below.

---

This is how the evaluation goes:

1. Dataset Cleaning

   This dataset contains cancellation, discounts and NaN customer ids. On top of it, it is truncated. For the sake of simplicity, discounts will be removed. I will need to clean the data for NaN customer ids and cancellation for purchases not within the dataset's timeframe

2. Dataset Splitting

   This dataset is split into calibration and holdout set, with 6 months long data each

3. Tranform Calibration Dataset

   The calibration dataset is transformed to obtain the input for the lifetimes model (frequency, recency, T, monetary value) and BSM (R: average monthly purchase).

4. Compute Actual CLV

   The holdout dataset is computed to obtain the actual CLV (aggregate purchase value of each customer)

5. Compute Predicted CLV

   The inputs from bullet 3 is used to predict the CLV value in the next 6 months using both of the models.

6. Evaluate Estimate

   RMSE is used to evaluate the difference in actual and predicted value for both model. It turns out that the BSM model has a lower RMSE compared to lifetimes.

   The values are:

---

Can anyone help me figure out why this is happening to lifetimes? Maybe I may have the wrong implementation? Or, the dataset does not work well with lifetimes? (If dataset is the problem, can you link me to a real dataset that works well with lifetimes?)

Please do add comments and changed in the notebook

[psygo]

Interesting stuff. The guys in the paper even mention the use of Monte Carlo Methods that would essentially do the same thing as the probabilistic models here, but with more accuracy; and which is something I've been wondering about for a while.

One thing you could try is to vary the Lifetimes's algorithm's penalizer and see if the RMSE does go down. The plots should give you some intuition if you're penalizer has a good value or not.

Frankly, the more I study the algorithms in the Liftetimes package the more I think they could either be replaced by somewhat simpler models or maybe get revamped with adaptations into more recent models, such as Neural Networks.

However, what should be highlighted here is that the formula you used is not recommended for Non-Contractual settings. This is even mentioned in the paper you linked (the BSM is inside section 3.1 entitled Models for Contractual Contexts). Bruce Hardie even has some notes on why that formula isn't appropriate.

Nonetheless, you could stick to the BSM depending on the use case I guess, i.e. if it does indeed work.

[psygo]

If you would like other datasets to try this on, take a look at the Lifetimes datasets folder. It has an artificial dataset, which is featured in the Quickstart page and part of the CDNOW dataset (around 1997), which is used in the literature.

[vierivincent15]

Hi, thank you so much for your help.

I have tried varying the penalizer coefficient, and at some value, the lifetimes model do actually perform better than the BSM model. Thank you for the recommendation.

And regarding your comment on simpler model, do you have any recommendations in mind on what model that is suitable for non-contractual context I can try experimenting with?

Lastly, do you think lifetimes will eventually perform better when fed with data that is more than 1 year long? (i.e. 2/3 year) This reason is that I will eventually use lifetimes to compute CLV with transactions data that is about 2/3 years. However, I am not that sure and can't directly test it because the UCI dataset and the CDNOW dataset are both only 1 year long.

[psygo]

I have tried varying the penalizer coefficient, and at some value, the lifetimes model do actually perform better than the BSM model. Thank you for the recommendation.

Cool that it helped.

And regarding your comment on simpler model, do you have any recommendations in mind on what model that is suitable for non-contractual context I can try experimenting with?

All of the models in the fitters folder of the Lifetimes library are supposed to deal with Non-Contractual settings. Try the BG/NBD at first, as it is one of the simplest.

Lastly, do you think lifetimes will eventually perform better when fed with data that is more than 1 year long?

There is no guarantee it will perform better with more customers or a longer period, but generally that's a good directive. Usually, we try to use data that comprehends at least one cycle of purchasing behavior. For example, if the store you're studying has a cycle of purchases that is only 3 months long, then anything bigger than 3 months should be minimally good enough.

[vierivincent15]

Thank you for the help. Greatly appreciate it :)