search

meeg-ml-benchmarks / brain-age-benchmark-paper

M/EEG brain age benchmark paper
https://meeg-ml-benchmarks.github.io/brain-age-benchmark-paper/
BSD 3-Clause "New" or "Revised" License
24 stars 10 forks source link

[WIP] Hyperparameter tuning and validation loss tracking for DL models #24

Open hubertjb opened 2 years ago

hubertjb commented 2 years ago

This PR builds on #23 to track validation loss and save history at the end of cross-validation. I also started a script to perform hyperparameter tuning with Optuna, but it's still in progress.

gemeinl commented 2 years ago

Hubert and I agreed to first make runs with default hyperparameters and the latest changes introduced in https://github.com/dengemann/meeg-brain-age-benchmark-paper/pull/27 and with this PR (dimensionality reduction for Cam-CAN).

The runs are: 4 datasets x 2 models x 2 decoding modes -> 16 runs

For Cam-CAN we will do additional 4 runs: 2 models, 2 decoding modes, dimensionality reduction

If these runs do not bring us into expected performance ranges, we will do hyperparameter optimization. Discussion is still ongoing whether we will do something simple as a grid search, something somewhat involved as optuna, or something even more involved as DEHB (developed by the machine learning lab in Freiburg, https://ml.informatik.uni-freiburg.de/wp-content/uploads/papers/21-IJCAI-DEHB.pdf).