search

reml-lab / mTAN

Code for "Multi-Time Attention Networks for Irregularly Sampled Time Series", ICLR 2021.
MIT License
113 stars 22 forks source link

Doubts about the results of Interpolation on PhysioNet #5

Closed DingShizhe closed 2 years ago

DingShizhe commented 2 years ago

The MSE score of interpolation performance on PhysioNet with 90% data points observed is 4.798 ± 0.036 ×10^−3. But I get 1.2 ×10^−3 after I run python3 tan_interpolation.py --niters 500 --lr 0.001 --batch-size 32 --rec-hidden 64 --latent-dim 16 --quantization 0.016 --enc mtan_rnn --dec mtan_rnn --n 8000 --gen-hidden 50 --save 1 --k-iwae 5 --std 0.01 --norm --learn-emb --kl --seed 0 --num-ref-points 64 --dataset physionet --sample-tp 0.9.

If the test data and prediction results are restored to the original scale according to the normalization of data preprocessing, then I get MSE of 4225.6895.

So how can I get the results of the paper presented?

DingShizhe commented 2 years ago

The result of the script is the mean prediction error of the whole instance, and the mean prediction error of interpolation points is presented in the article. I've reproduced it.

adaptdeng commented 2 weeks ago

The result of the script is the mean prediction error of the whole instance, and the mean prediction error of interpolation points is presented in the article. I've reproduced it.

Thank you for sharing your experience and findings! I've been working on reproducing the results as well and have encountered similar challenges. However, my results (around 3.7 ^ 10-3) are still higher than the reported results. I mainly changed the evaluation metrics to only consider output the MSE for interpolated points.

Did you find any subtle implementation details that significantly affected the results? For example, does the decoder part also only see the subsampled_tp? If possible, can you share your code snippet for the training loop?

Thank you for your help in advance!