search

Rose-STL-Lab / dyffusion

[NeurIPS 2023] A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting
https://salvarc.github.io/blog/2023/dyffusion
Apache License 2.0
124 stars 12 forks source link

Some Difference in Reproduced Results #5

Open JaydenLyh opened 2 weeks ago

JaydenLyh commented 2 weeks ago

The paper and the code are both very well done and have been very helpful for my research. I particularly appreciated the detailed explanations and the thoroughness of the experiments. I cloned this repository and trained the dyffusion, but I got some difference in reproduced results. Steps Followed I followed the steps provided in the README to train the model: First stage: Train the interpolator network python run.py experiment=spring_mesh_interpolation Second stage: Train the forecaster network Encountered some issues, so I commented out lines 229-231 in wandb_api.py: if os.path.exists(ckpt_path) and ckpt_path != best_model_fname: os.remove(ckpt_path) # remove if one exists from before os.rename(best_model_fname, ckpt_path) Then ran: python run.py experiment=spring_mesh_dyffusion diffusion.interpolator_run_id=<WANDB_RUN_ID> Testing： python run.py mode=test logger.wandb.project=DYffusion-spring-mesh logger.wandb.id=<run_id>

Actual Results test/50ens_mems/avg/crps = 1.54671955108642 test/50ens_mems/avg/mse = 36.7373008728027 test/50ens_mems/avg/ssr = 2.02742147445678

Results on Navier Stokes Dataset On the Navier Stokes dataset, I was able to reproduce results similar to those reported in your paper. Could you please provide guidance on what might be causing these discrepancies? Any help or suggestions would be greatly appreciated. Thank you

salvaRC commented 1 week ago

Hi and thank you for the feedback! I'm glad that you found our work and code helpful and happy to hear that the navier stokes results are as expected.

I am sorry to hear about the issue with the spring mesh dataset. To better understand it, could you share with me the configs (i.e. hydra_config.yaml files) for both the interpolation and dyffusion training stages that you use?

JaydenLyh commented 1 week ago

springmesh_configs.zip Thank you so much for your response! I've uploaded the files you requested. Please let me know if there's anything else you need or if there are any additional details I can provide. Thanks again for your support!