search

Novartis / torchsurv

Deep survival analysis made easy
http://opensource.nibr.com/torchsurv/
MIT License
57 stars 7 forks source link

How to Calculate Integrated Brier Score (IBS) for a CNN Model with torchsurv? #57

Open tadongguk opened 2 days ago

tadongguk commented 2 days ago

I'm currently using the torchsurv library for survival analysis with a CNN model, but the library only supports calculating the C-index and does not provide the Brier score (IBS). 

We evaluate the predictive performance of the model using

the [concordance index](https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.cindex.html) (C-index), which measures the the probability that a model correctly predicts which of two comparable samples will experience an event first based on their estimated risk scores,

the [Area Under the Receiver Operating Characteristic Curve](https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.auc.html) (AUC), which measures the probability that a model correctly predicts which of two comparable samples will experience an event by time t based on their estimated risk scores.

We cannot use the Brier score because this model is not able to estimate the survival function

What would be the best approach to calculate the Integrated Brier Score (IBS) for a CNN model?

Could I use another library like scikit-survival or lifelines for this, and if so, how could I integrate those with my CNN-based model? Additionally, if using the Weibull model in torchsurv, is it necessary to use the Weibull model to compute IBS, or are there other models I could use?

melodiemonod commented 2 days ago

Dear @tadongguk,

Thank you for your interest in torchsurv and your question.

1/ To compute the IBS, one needs to be able to estimate the survival function. This is not library-dependent, it is the definition of the metric, see notes here https://opensource.nibr.com/torchsurv/_autosummary/torchsurv.metrics.brier_score.html and relevant reference. The survival function is available for the Weibull Accelerated Time Failure (AFT), but not directly for the Cox Proportional Hazards model.

2/ The NN architecture (in your case a CNN) is defined on the survival model's parameters - the log relative hazard for the cox proportional hazards model, and the Weibull parameters for the Weibull AFT model. In conclusion, you can use your CNN to estimate the Weibull parameters for the Weibull AFT model, and get an IBS.

tcoroller commented 1 day ago

Hi @tadongguk!

Thank you for using TorchSurv!

In addition, I would suggest to checkout our tutorial that covers all metrics (including Brier score) and how to use it.

TL;DR:

  1. You need to train a Weibull model, your model will output two (2) values: one for the shape and one for the scale. Your input and architecture in between can be anything, thats the beauty of TorchSurv 🔥
  2. Once trained, compute the survival function for the Weibull log parameters.
  3. Compute the Brier score using the survival function, events and times such as brier_score(surv, event, time)