Question for reproducing benchmark models in moving MNIST

[Pachark]

Hi,

Thanks for your work.

I reproduced the PhyDNet model with default parameters in OpenSTL package (https://github.com/chengtan9907/OpenSTL). I use your Moving MNIST dataset with 9000 training samples, 1000 test samples. To be consistent with your configs, train_test_split with random_state as 123 is used to split another validation set. But I got better results than reported in your paper.

Reproduced: mse:53.35, mae:107.25, ssim:0.90. Paper: mse:58.70, mae:124.1, ssim:0.8350.

Are there some configs I need to change to reproduce the PhyDNet in moving MNIST dataset?

[gaozhihan]

It's wonderful that you were able to achieve improved results with PhyDNet. There are many experimental details, particularly regarding the optimizer, that can impact model performance. For example, the original PhyDNet implementation utilizes a ReduceLROnPlateau lr scheduler, but this does not seem to be present in the OpenSTL repository implementation.

[Pachark]

Thanks for you reply!

I found another problem about the metric SSIM. The metrics reproduced by OpenSTL seem to be inconsistent with Earthformer paper. Even when the MSE and MAE are close to Earthformer paper, the SSIM of OpenSTL seems better.

Reproduced metrics of PhyDNet in OpenSTL: mse:53.35, mae:107.25, ssim:0.90. Metrics of PredRNN in Earthformer paper: mse:52.07, mae:108.9, ssim:0.8831.

Did I miss anything that may cause the inconsistency?

[gaozhihan]

The trends of MSE, MAE, and SSIM metrics may not necessarily be consistent with each other. Each metric captures different aspects of reconstruction performance. Otherwise we would only need one of them in our paper instead of reporting scores in all three metrics.

[Pachark]

Thanks. I'll close the issue.