search

environmental-forecasting / model-ensembler

Model Ensemble tool for batch workflows on HPCs
https://pypi.org/project/model-ensembler/
MIT License
13 stars 1 forks source link

Hyperparameter tuning #18

Open MartinSJRogers opened 3 years ago

MartinSJRogers commented 3 years ago

Potentially a major time-saver when training a convolutional neural network. I estimate I will be at the stage of training a CNN in about 4 weeks, so it would be good to find a time to speak in November about this. I'm assuming this is something others e.g. @tom-andersson would be interested in?

JimCircadian commented 3 years ago

@MartinSJRogers Absolutely, I think integrating this with model code is not difficult so we can run through an example together and I can get a steer if there are more complicated workflows required as well. Thanks!

JimCircadian commented 2 years ago

Speaking with Martin, WandB offers real opportunity to integrate with model ensembler to parameterise/manage sweep jobs, as the current method is suboptimal for scaling/larger jobs. What we need to look at is the submission strategy of the wandb agent running scripts against an active ensembler which can then throttle and manage the jobs.

Interestingly, wandb in the simple sense runs linearly but I reckon that's because we've not explored the full extent of the controller mechanism. Have a look at that and the integration posssibilities.

Current sweep jobs are single long running jobs on the HPC, which won't scale...

Might be of interest given your testimonial gathering @scotthosking

JimCircadian commented 2 years ago

Clones #16