search

hedixia / HeavyBallNODE

13 stars 2 forks source link

5.3. time series Experiments issue #1

Open Seunghyeon-Cho opened 2 years ago

Seunghyeon-Cho commented 2 years ago

Hi, Thanks a lot for the brilliant repo!

I ran the experiment in 5.3 experiments (Learning dynamical systems from irregularly-sampled time series Experiments) and the results are strange. I ran both NODE and HBNODE. According to the paper, the loss of HBNODE should be lower, but the train / val / test loss of NODE is lower.

Is it possible to check what's wrong with the code?

hedixia commented 2 years ago

This finalized version is slightly different from the version used in paper, which perhaps is the reason that produces different results. I will look into the difference and see what causes the issue.

Seunghyeon-Cho commented 2 years ago

Thank you.

According to the paper, HBNODE should be better than NODE in loss value, but in the opposite case (NODE is better than HBNODE), so I left an issue.

Let me know if you find anything!

Seunghyeon-Cho commented 2 years ago

I have one more question.

In the Walker2D kinematic simulation experiment, is HBNODE taking longer training time than NODE a normal result?

If there is a checkpoint for the experiments, I wonder if you can share it. It seems that the model is heavily influenced by external influences such as random seeds.

hedixia commented 2 years ago

for 5.3 i think the problem is the preshrink parameter is not the same for different tasks. I think in our experiments they were 0.1, but in this final version i forgot to include that. In walker 2d, we did 3 tries and all of them HBNODE is better in accuracy and taking longer time as well. Considering the limited resources i have right now, it'll take quite a long time to run that experiment again. I will try to find if i have the checkpoints, but it's almost a year and I am not sure whether I could find them.

Seunghyeon-Cho commented 1 year ago

Hello.

It's been a long time, but I'm asking again. I wonder if the results of one experiment in 5.3 solved the problem that cannot be restored.

Thank you.