search

Rose-STL-Lab / dyffusion

[NeurIPS 2023] A Dynamics-informed Diffusion Model for Spatiotemporal Forecasting
https://salvarc.github.io/blog/2023/dyffusion
Apache License 2.0
148 stars 18 forks source link

The baseline reproduction result is wrong #6

Open lmllmllm opened 3 months ago

lmllmllm commented 3 months ago

OISST dataset: DDPM test results for SST: test/50ens_mems/avg/mse=1211.059 test/50ens_mems/avg/crps=24.399 test/50ens_mems/avg/ssr=2.479 The results of the two repeated experiments are almost the same But for Dyffusion and MCVD, I can reproduce the results similar to those in the paper according to your code I would like to ask how to reproduce the results of DDPM Secondly, do you have the corresponding codes for the other two baselines Dropout and Perturbation in your project? I am a beginner and hope to get your help.

salvaRC commented 3 months ago

Hi and thanks for your comments!

To better understand why this is happening, could you share the config that you're using (i.e. the hydra_config.yaml file)? I must say that I didn't re-run the DDPM baseline after cleaning up the code, so the default hyperparameters might not be optimal. For example, using fewer diffusion steps (e.g. diffusion.timesteps=5) was important for some datasets. Did you do this?

For the dropout baseline, use the <dataset>_time_conditioned.yaml configs in configs/experiment/. For the perturbation baseline, simply train a deterministic model (module.enable_inference_dropout=False) and run inference/tests settingmodule.prediction_inputs_noise=EPS for EPS>0.

`

lmllmllm commented 3 months ago

Thank you for your reply, this is my configuration hydra_config-v2.zip