Closed NataliaVConnolly closed 6 years ago
Hi there, Thanks for the kind words.
Your actual TTE might be the censored TTE (so pred_tte would seem to overestimate it for the censored observations). Try removing censored values to get a sense of magnitude.
It looks like your trying to predict the median TTE (a * np.power(-np.log(1.0 - p), 1.0 / b)
in numpy with p=0.5). I would be more interested in about 50% of the actual tte being below the predicted than how close it is
If you want to get a general sense of performance, take the (scale dependent) correlation between actual and predicted. Also try (scale independent) correlation after dense rank (leading to rank-correlation)
I think the problem is still actual because in example with full pipeline: https://github.com/ragulpr/wtte-rnn/blob/master/examples/data_pipeline/data_pipeline.ipynb there is picture for one individual, which is not trending right TTE:
And there are not so many censored data:
Could you know what could be the reason for that and what could be fixed in the example to fit data appropriately?
So @aprotopopov the direction it goes actually makes sense, given the information available. If someone hasn't committed in a while there's a risk that it's a long break or that they've stopped altogether. I bet that's what the algo thinks anyway.
The scale is a bit off though. It's worth digging into, but it's worth considering that that is a serial-committer and many in the dataset commit very sparsely (high TTEs)
Another way I did not mention if you're worried about performance is to see how good it is in terms of fixed windows:
from sklearn import metrics
aucs =[]
# compare score to sliding box up to some width, up to last decile
max_box_width = np.sort(seq_lengths)[-len(seq_lengths)/10]
for box_width in xrange(max_box_width):
if (box_width%10)==0:
# select only unmasked and comparable datapoints.
m = ~np.isnan(y[:,:,1])
# uncensored or within box_width of boundary
m[m] = (y[:,:,1][m]==1)|(box_width<y[:,:,1][m])
actual = y[:,:,0][m].flatten()<=box_width
pred = weibull.cmf(a=predicted[:,:,0],b=predicted[:,:,1],t=box_width)[m].flatten()
fpr,tpr,thresholds = metrics.roc_curve(actual,pred)
print 'auc: ',metrics.auc(fpr,tpr),' sliding box ',box_width
aucs.append(metrics.auc(fpr,tpr))
plt.plot(aucs)
For git-repos I get AUCs in the 70-90 range for most reasonable windowsizes
Btw @aprotopopov I did not forget about your PR, it's great but I'm just trying to merge it with some local changes! Will comment ASAP
Hi Egil,
Thanks so much for releasing the end-to-end wtte-rnn code in data_pipeline! Very cool stuff.
I had a question about validating the performance though. To check how well the model predicts TTE, I did the following:
where
What I got looked like this:
So if you plot predicted TTE vs. actual they don't agree much, not even directionally. Clearly I am missing something. Is this not a valid way to compare predicted vs. actual TTE?
Thank you! Natalia