Create a benchmark based on PDEBench

[Deathn0t]

Let's define a benchmark suite based on the PDEBench: https://github.com/pdebench/PDEBench

[Deathn0t]

@iamyixuan advanced on this here: https://github.com/deephyper/benchmark/tree/main/lib/PINNBench/Diffusion-reaction

Next, we need to

• [x] record learning curves metadata
• [ ] add skip connection capabilities
• [x] add dataset selection capability DEEPHYPER_BENCHMARK_DATASET=...

[Deathn0t]

• [ ] document the possible values for DEEPHYPER_BENCHMARK_DATASET, what happens if another dataset is used? it looks like the data generation is happening in the installation.
• [x] for the dataset document the storage size of the dataset
• [x] fvcore.nn is not part of the installation procedure (not inside the requirements.txt) is it normal?
• [x] the flops could be an additional objective in MOO
• [ ] would be great to have more parameters to tune such as:
  • regularization hyperparameters
  • [x] dropout_rate
  • [x] early_stopping_patience
  • [ ] batch_norm (batch normalization)
  • [x] weight_decay
  • [x] skip_co for skip connections (can be just a boolean with "regular skip co")
  • [ ] stochastic weight averaging
  • [x] optimizer

[iamyixuan]

How to check the checkboxes above? Tried many times and only got to check two.

[Deathn0t]

• [ ] make weights as hyperparameters