Unstable c-index results

[JiayuLi-997]

Hi. Many thanks for this powerful implementation of survival analysis. I am new to survival analysis, and I am trying to use the Cox-Time model for my dataset. But I observed the c-index result on the test set is really unstable with different random seeds. (varying from less than 0.6 to more than 0.7) Did you find the same problem in your experiments? Or do you have any suggestions about it?

[havakv]

Hi! Yes I've found the c-index can be quite unstable. However, for large datasets (such as the KKBox dataset) it gives more stable results. If you're working on small datasets, you probably can't trust the c-index results that much, and if you're doing cross-validation the variability between the folds is likely larger than than the variability between the models. There's no easy way around this sadly.

One way to compare models is to use different seeds and check which models gets the best c-index for each seed. I've found that often the best model is best for many seeds, even thought the c-index varies a lot between seed. Using this evaluation, you essentially use the c-index to rank your models, but you don't really care about the value of the c-index.

[JiayuLi-997]

Thanks for your patient and timely reply!

Actually, I am applying the time-varying cox models in another scenario (to be specific, for user churn analysis in the video game). And I am trying to find the best model, covariates, and parameters to fit my problem.

Unfortunately, the best models are not always the same for different seeds. Do you think it is reasonable to choose the model with the least loss on validation set?

And I also wonder why the model with significant smaller validation loss may perform worse on C-index (on validation set). In my opinion, the loss and C-index should be consistent, according to their calculation. Have you noticed the similar phenomena?

[havakv]

Which model selection criterion you use should probably be reflected by how you want to apply your model in practise. So if you only want to rank your users compared to each other (e..g, report the highest 10%), a high concordance is important. However, if you want to give probabilities for churn for the users, concordance is likely not the best scoring rule and something like the log-likelihood (validation loss) might make more sense. For example, if you divide all your survival predictions by 10, you concordance is unchanged while your loss should be much worse.

Ideally, I should probably encourage you to use multiple evaluation metrics, but I agree that it's probably fine to just consider the validation loss.

If I recall correctly, a theoretically optimal model (perfect survival estimates) should obtain an optimal concordance and loss. However, all other models might have a tradeoff between the two. For instance, DeepHit is typically good at ranking (high concordance) but often produce very bad survival estimates (which would give a bad survival likelihood).

[JiayuLi-997]

Thanks a lot for your comment! It really helps a lot.

I read more works about survival analysis this week. You are right about the difference between likelihood and concordance. And in our application, I think it is better to choose models based on likelihood.

Thanks again!

[havakv]

Happy to help!