Tracking experiments to benchmark MOO

[Deathn0t]

Continuous Benchmark Functions

DTLZ (10 random seeds each for a fixed number of evaluations):

• [ ] Random
• [x] NSGAII (pymoo)
• [x] Parmoo (local search solver)
• [x] DeepHyper Scalarizations

Hyperparameter Optimization Benchmarks

Community Benchmarks

HPOBench (10 random seeds for each, fixed number of evaluations, 2-objectives: valid-rmse, training time):

• [x] navalpropulsion
  • [x] DH/Random
  • [x] DH/Chebychev
  • [x] OPT/NSGAII
  • [x] OPT/TPE
• [ ] parkinsonstelemonitoring
  • [ ] DH/Random
  • [ ] DH/Chebychev
  • [ ] OPT/NSGAII
  • [ ] OPT/TPE
• [ ] proteinstructure
  • [ ] DH/Random
  • [ ] DH/Chebychev
  • [ ] OPT/NSGAII
  • [ ] OPT/TPE
• [ ] slicelocalization
  • [ ] DH/Random
  • [ ] DH/Chebychev
  • [ ] OPT/NSGAII
  • [ ] OPT/TPE

JAHS-Bench (5 seeds for each, 3-objectives: valid-acc, latency, an size_MB, 3 datasets: Cifar10, Colorectal, Fashion-Mnist, 8 + 1 hyperparameters):

• [ ] Fashion-Mnist
  • [ ] Random
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
  • [ ] DeepHyper-Chebychev
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
  • [ ] Optuna/NSGAII
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
  • [ ] Optuna/TPE
    • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
  • [ ] Parmoo (local search solver)
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes

Custom Benchmarks

ECP-Candle/Pilot1/Combo:

• [x] Random
  • [x] 10 nodes
  • [x] 40 nodes
  • [x] 160 nodes
• [x] DeepHyper-Chebychev
  • [x] 10 nodes
  • [x] 40 nodes
  • [x] 160 nodes
• [x] Optuna/NSGAII
  • [x] 10 nodes
  • [x] 40 nodes
  • [x] 160 nodes
• [x] Optuna/TPE
  • [x] 10 nodes
  • [x] 40 nodes
  • [x] 160 nodes

PINNBench/DiffusionReaction:

• [ ] Random
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
• [ ] DeepHyper-Chebychev
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes
• [ ] Optuna/NSGAII
  • [ ] 10 nodes
  • [ ] 40 nodes
  • [ ] 160 nodes