Automated hyperparameter tuning

[s-sd]

Subject of the feature

What are you trying to do and how would you want to do it differently?

At MICCAI I saw a lot of interest in automated hyperparameter search/ tuning. The feature is available in several libraries but integrating it with parameters specified in the config file and with the deepreg train function is not straightforward. I suggest that we add some simple functionality that allows for an automated search procedure.

To start out we could add a simple function called something like tune_and_train which takes in a slightly modifed version of the config file. The modified config file can allow the user to specify either a range for a certain parameter or multiple values for a paramter. Then using random serach the tune_and_train function first selectes the best hyperparameters from its search and then uses these to train the network for a user defined number of epochs. To start out, we can offer this random search procedure which runs the train until a user defined number of epochs for each random combination of parameters that are selected from the user defined space for the parameters. Then the validation accuracy is used from each run to select the best combination which is then used to train the final model for a user defined number of epochs. As a starting point we can offer this functionality for a few parameters in the config file like batch size and number of channels. Then we can extend with other parameters and also other search algorithms, as required.

(Copied from Issue #424 )

Is it something you currently you cannot do?

Yes

Is this related to an issue/problem?

Originally suggested in Issue #424 comments thread

Has the feature been requested before? If yes, please provide a link to the issue.

No

If the feature request is approved, would you be willing to submit a PR? (Help can be provided if you need assistance submitting a PR)

Yes

[mathpluscode]

Tensorflow seems to have this https://www.tensorflow.org/tensorboard/hyperparameter_tuning_with_hparams we could have a look to see if it's easy to integrate.

[mathpluscode]

BTW, last time I tried some auto lr tuning with pytorch lightning https://pytorch-lightning.readthedocs.io/en/latest/lr_finder.html, the experience is not amazing tho. @sbb-gh do you know some other relevant functions ;) ?

[YipengHu]

BTW, last time I tried some auto lr tuning with pytorch lightning https://pytorch-lightning.readthedocs.io/en/latest/lr_finder.html, the experience is not amazing tho. @sbb-gh do you know some other relevant functions ;) ?

why not amazing?

[mathpluscode]

BTW, last time I tried some auto lr tuning with pytorch lightning https://pytorch-lightning.readthedocs.io/en/latest/lr_finder.html, the experience is not amazing tho. @sbb-gh do you know some other relevant functions ;) ?

why not amazing?

I was imagining some auto-tuning during the training lol... Well maybe it's my expectation problem. The tool was just choosing a lr with some data before the training, and I'm not confident about the chosen lr.

[sbb-gh]

@mathpluscode Apologies for the late reply. I've only tried some neural architecture search on Tensorflow, where we tune the network architecture. It was a nightmare to use.

[sbb-gh]

Maybe just having a systematic approach would be good enough.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mathpluscode]

We did encounter many problems in demo with the loss weights, thus this issue is still important.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[s-sd]

@mathpluscode @YipengHu I would be quite interested in working on this. Is it okay if I assign myself for this issue?

[YipengHu]

@mathpluscode @YipengHu I would be quite interested in working on this. Is it okay if I assign myself for this issue?

Thanks! Can we have a quick discussion on this? I'm aware @mathpluscode and I might have some diverging opinions on this? ;)

[YipengHu]

@s-sd do you have a reference for the work you mentioned?

[s-sd]

@YipengHu yes, we can have a discussion. Do you mean a reference for the MICCAI works exploring hyper-parameter tuning?

[mathpluscode]

@s-sd 😁any updates? Some papers we can check?

[s-sd]

@s-sd grinany updates? Some papers we can check?

@mathpluscode @YipengHu I am listing some titles for papers from MICCAI 2020. I think the best way to access them all in on place would be on the MICCAI 2020 pathable portal. If you can't access the portal, let me know and I'll try to find alternative springer or arxiv links for these.

• Auto-weighting for Breast Cancer Classification in Multimodal Ultrasound
• AutoSNAP: Automatically Learning Neural Architectures for Instrument Pose Estimation
• Searching Collaborative Agents for Multi-plane Localization in 3D Ultrasound
• NAS-SCAM: Neural Architecture Search-based Spatial and Channel Joint Attention Module for Nuclei Semantic Segmentation and Classification
• Neural Architecture Search for Optimization of Spatial-temporal Brain Network Decomposition
• MS-NAS: Multi-Scale Neural Architecture Search for Medical Image Segmentation

A few other resources that I found useful (including some tools that have good documentation and provide automated tuning and also including some resources that you've shared with me previously):

• https://arxiv.org/abs/1603.06560
• https://arxiv.org/abs/2101.01035
• https://docs.ray.io/en/master/tune/tutorials/tune-advanced-tutorial.html
• https://papers.nips.cc/paper/2011/file/86e8f7ab32cfd12577bc2619bc635690-Paper.pdf
• http://proceedings.mlr.press/v28/bergstra13.pdf
• http://hyperopt.github.io/hyperopt/#algorithms
• https://keras-team.github.io/keras-tuner/

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.