havakv
commented
2 years ago Hi, and thank you for the kind words.
What you ask for is not available in this package, and I don't think I will prioritize this going forward (sorry about that). However, if you (or someone else) wants to have a go at it, please make a PR as I'm sure there are many more that wants this functionality.
I think that the bootstrap you suggest is a good substitute for obtaining confidence intervals, although it will only work for small datasets. I guess the size of you bootstrap will depend on how computationally expensive your computations are. You might be able to get away with fewer samples by considering the pairwise differences in performance for each sample. I often see that one method always performs better than another in each sample, but the variability between the samples makes the results seemingly insignificant. I don't have any resources on this, but just something you might want to explore.
The hazard ratio have been discussed in a couple of other issues, so you can have a look at e.g. #96 and let me know if there is anything you don't understand. In short, I don't think it is particularly relevant for models that are not proportional hazards.
For calibration plots, it should be fairly easy to just dump a csv with predictions and true values an plot them in R. If you really want to use python for this, I guess it shouldn't be too hard to implement (depending on your representation of the calibration). A couple of years ago I did this for the Cox-Snell residuals, which was fairly straight forward. I can't find the code for this now, but you might want to give it a go.
I'm sorry that I can't be of more assistance, but the sad truth is that python will never have all the statistical tools available in R, so now and then you might have to use some R code if you don't want to reimplement it in python. Also, I no longer have that much time to continue development of this package, so I don't think many new features will be included, unless there is a community effort.
kyuchoi
Hi, first of all, thank you so much for your great job!
I have two requests in Pycox:
1) Is it possible to get 95% confidence interval of time-dependent C-index in Pycox package? I think it would be possible using Jackknife method as follows: https://www.jstatsoft.org/article/view/v015i01
I think it only might be achievable via training models a thousand times or more as follows, which requires a lot of training time: https://machinelearningmastery.com/calculate-bootstrap-confidence-intervals-machine-learning-results-python/
However, I think the training time usually not that long for just tabular data, not large image data.
However, these medical statistics usually use R package, which is not Python, so it is quite annoying. Moreover, reviewers usually require 95% confidence interval of C-index (as well as hazard ratio), and I was wondering if the Pycox has this extension as well.
2) Is it possible to get calibration plots in Pycox package? Currently, additional R coding is required to get calibration plots.
Thank you again! Best