how to calculate estimated risk

[ad05bzag]

Hi, I was asked to calculate the following metric: https://scikit-survival.readthedocs.io/en/latest/generated/sksurv.metrics.concordance_index_censored.html which requires 3 input arrays:

• event_indicator - Boolean array denotes whether an event occurred
• event_time – Array containing the time of an event or time of censoring
• estimate – Estimated risk of experiencing an event

how would i calculate estimated risk from a, e.g. DeepHitSingle model? or alternatively from any Cox models

edit: for CoxPH with 1 out feature, model.predict() on test data results in an identical value between ev.concordance_td('antolini') and sksurv.metrics.concordance_index_censored()

[havakv]

So, the Antolini concordance index should be equal Harrell's (sksurv) although handling of tied event times might vary between implementations. So your observations are correct. For proportional hazard models, such as CoxPH, the concordance (sksurv) is a very nice metric as it is simple to define an "estimated risk" from the output of the models (the hazards are proportional so the survival estimates cannot cross each other). For non-proportional models, such as DeepHit, CoxTime, LogisticHazard etc, is it is not as obvious how to define the "estimated risk" from the model. Note that this problem is because of the evaluation criteria (concordance) and not the models. The Antolini concordance is one attempt to address this, as it considers the full survival function and not just a single number per individual. We have written something about this in Section 4.1 in this paper.

If you still want to use the Harrell's concordance index (sksurv) you need to defined a "risk estimate". That can for instance be the survival estimate at one specific time (such as 50 sec), or the integrated survival function (if you use one of these, you should probably invert them to get a risk estimate and not a survival estimate). If you use CoxPH both of these will give the same concordance as what you got, but of the non-proportional methods your results may vary.

I know this probably wasn't the answer you were looking for, but I believe it is an important one as the Harrell's concordance index (sksurv) has some serious issues.

If you have problems getting the sksurv concordance to work, just ask here again and I'll give you some examples.

[ad05bzag]

Dear @havakv your answer was great! It is insightful and it helped me develop a more intuitive understanding of survival metrics. Much better than "use method A for that" :)

I need to work through Antolini's formulation of concordance index to understand how it takes into account the full survival function.

I made sksurv work the following way. But I am also curious to see some of the examples you mentioned

do_nothing = [(col, None) for col in df.columns if col not in ['death', 'time']]
x_mapper = DataFrameMapper(do_nothing)
get_target = lambda df: (df['time'].values, df['death'].values)

x_test = x_mapper.transform(X_test).astype('float32')
durations_test, events_test = get_target(X_test)

model = CoxPH(net, tt.optim.Adam) 
model.optimizer.set_lr(0.005)

## model test 
    surv = model.predict_surv_df(x_test)
    ev = EvalSurv(surv, durations_test, events_test, censor_surv='km')
    ci_td_test = ev.concordance_td('antolini')
    ci_trad_test = sksurv.metrics.concordance_index_censored(events_test.astype('bool'), durations_test, model.predict(x_test).flatten())

ci_td_test and ci_trad_test are the same

[havakv]

So in you example above, you can choose to evaluate the concordance at a specific time represented by the index idx in the code below. You can then do

surv = model.predict_surv_df(x_test)
concordance_index_censored(events_test.astype('bool'), durations_test, 1-surv.iloc[idx])

Fox CoxPH, this should give the same concordance as your example above, meaning the results should not be dependent on idx (don't use idx = 0). For non-proportional models, on the other hand, the results depend on idx.

The next obvious question would then be "how do I choose idx"? To which my answer again is that it is the Harrell's concordance that is the problem and not the idx. If you use a metric such as the Antolini concordance, your CoxPH model will have the same concordance as for Harrell's concordance, and you don't have to choose idx for the non-proportional methods.

[havakv]

You can define other function of the survival estimates that represent a risk function, but I haven't seem much research on this topic. The only thing that comes to mind is the mortality used for random survival forests (see section 5).