Missing yet described k_fold_cross_validation functionality

[lotrus28]

Hello! I am using v 0.25.4 of lifelines I have recently started using lifelines and saw this in the documentation:

lifelines.utils.k_fold_cross_validation(fitters, df, duration_col, event_col=None, k=5, scoring_method='log_likelihood', fitter_kwargs={}) ... predictor (string) – a string that matches a prediction method on the fitter instances. For example, predict_expectation or predict_percentile. Default is “predict_expectation” The interface for the method is: predict(self, data, **optional_kwargs)

The same is written as a commentary in the library code under k_fold_cross_validation's definition.

But when I tried feeding predictor="predict_expectation" into k_fold_cross_validation I got this error:

Traceback (most recent call last):
  File "C:\Bin\Python36\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-22-7349d199dd2f>", line 7, in <module>
    fitter_kwargs=dict(predictor='predict_expectation'))
  File "C:\Bin\Python36\lib\site-packages\lifelines\utils\__init__.py", line 772, in k_fold_cross_validation
    fitter.fit(training_data, duration_col=duration_col, event_col=event_col, **fitter_kwargs)
  File "C:\Bin\Python36\lib\site-packages\lifelines\utils\__init__.py", line 54, in f
    return function(model, *args, **kwargs)
TypeError: fit() got an unexpected keyword argument 'predictor'

Apparently neither the function, nor any downstream functions (score, fit) not have a predictor parameter. I started looking around the lifelines code and then discovered some other issues. The documentation also contains mentions of the evaluation_measure parameter, which should allow users to create custom quality metrics different from concordance_index and log_likelihood, as long as that metrics can accept (event_times, predicted_event_times, event_observed) as arguments. But using this parameter produces a similar error:

TypeError: k_fold_cross_validation() got an unexpected keyword argument 'evaluation_measure'

On a side note, the k_fold_cross_validation code also contains this line:

testing_columns = df.columns.drop([duration_col, event_col])

But testing_columns variable is not used anywhere. It seems to be a leftover either from earlier versions, or a reminder for a feature yet to be implemented.

This was quite a letdown, when I discovered this. I trusted the documentation and expected to just plug in a function that assigns a quality score to a CPH model based on how many times it guessed right that a person survives for more than a week or a month. But neither can I do this, nor can I specify the type of prediction (median, expected...) used to calculate this score.

I think I will need to write a custom_k_fold_cross_validation function to implement my idea. But please clarify: were the predictor and evaluation_measure present in some earlier versions of lifelines? Or were these features announced in the documentation not so long ago and you are eventually going to implement these. And is the testing_columns variable indeed just a dangling end, or did I miss something important?

[lotrus28]

I see now, that k_fold_cross_validation maintained the arguments predictor and evaluation_measure in 0.23.9, but after 0.24.0 - 2020-02-20 they disappear. What is the reason for this?

[CamDavidsonPilon]

Thanks for bringing this up @lotrus28

This part of the code is the least developed unfortunately. I'd be happy to see improvements/new functions added back to lifelines if you end up writing something. Honestly, I would be happy with as much as an overhaul to the API.

Otherwise, I'll dig into this for a future release, maybe in a few weeks.

[lotrus28]

@CamDavidsonPilon Thank you for the quick response! I decided not to write anything new and to just downgrade to 0.23.9, since it has all the functionality that I need. It works fine, I guess.

I was just wondering if there was any significant reason you decided to ditch predictor and evaluation_measure from k_fold_cross_validation. Those options are extremely useful, but if there was a bug prior to 0.24.0, I think we should know.

To give some context on why I needed evaluation_measure: I wanted to select the best model based on how accurately it predicts 1 year survival status in CV.

[CamDavidsonPilon]

This issue has useful bugs and information for me to improve lifelines, so I'd like to keep it open