search

bachmannpatrick / CLVTools

R-Package for estimating CLV
54 stars 14 forks source link

How to evaluate fit performance? #150

Closed dsolito closed 3 years ago

dsolito commented 4 years ago

First, let me thank you for the package. The field (pCLV) is relatively new to me so my questions can seems dumb :

If i take this row, what are your observations?

cust6440 | 2018-04-01 | 2019-03-31 | 52.14286 | 3 | 468.130 | 0.60943445 | 1.51645987 | 0.350591385 | 172.10841 | 60.3397275

Thank you for your help!

David

pschil commented 4 years ago

Let me first give a a brief explanation how CLV is defined for the latent attrition models in this package:

As you can see, CLV is from estimation end (T) until infinity, and accounts for the probability to be alive (S), the number of future transactions (t(t)) and discounts all of this (d) (so that less and less emphasize is put on timepoints further away).

The expression in the integral is called DERT while the E(Spending) is the mean spending per transaction and assumed to be constant in the future. CLV therefore is calculated as DERT * predicted.mean.spending.per.transaction. DERT is an output form the pnbd() model and predicted.mean.spending is from the gg() model which by default is automatically fit when predicting.

See ?predict.clv.fitted.transactions for an explanation of what the different columns in the predictions are.

As you can see from the above definition, CLV as a concept is hard to evaluate because you cannot actually observe a customer until infinity (not even customer "death").

But you can evaluate it on the holdout data to get an impression of how good your estimate is. You can compare (1) the number of predicted transactions vs actual number of transactions or (2) the predicted value of a customer during the holdout period vs actual value. (1) The predicted number of transactions in the prediction period is CET which can be compared against actual.x, the number of observed transactions during the prediction period. (2) The predicted number of transactions predicted mean spending per transaction (CETpredicted.mean.spending = customer's predicted value in the prediction period) can be compared against actual.total.spending.

Based on these predicted vs actuals you can calculate popular performance measures such as RMSE, MAE, MAPE etc which each of course have their own strength and weaknesses.

To compare PAlive vs actually being alive (actual.x > 0) is a little more tricky. Something like a ROC curve / AUC can work for this.

Is the pCLV value estimated by week (if time.unit is by "week") ?

No, CLV is a monetary value that the customer is assumed to have when standing at the end of the estimation period, independent of the holdout period and not defined by any time unit.

dsolito commented 4 years ago

Hi @pschil ,

Thanks for your feedback! I will take a little time to reply with some performance evaluation exploration. As this could be a good use case for newbie like me...