Refactor SLURM functionality - Githubissues

earthnet2021 / earthnet-models-pytorch

EarthNet Models PyTorch - Deep Neural Networks for geospatial vegetation forecasting.

https://www.earthnet.tech/

MIT License

29 stars 6 forks source link

Refactor SLURM functionality #24

Open vitusbenson opened 1 year ago

vitusbenson commented 1 year ago

SLURM functionality is very important for larger model runs on clusters. However, currently it is just a bash script which needs to be adapted by each user, which is of limited usefulness.

Goal:

Be able to specify SLURM options in the config file and then just run with the normal python train.py configfile.yaml, which automatically starts and collects the slurm jobs.
Documentation for this

Requirements:

Being able to run slurm on new cluster:
- [ ] Training
- [ ] Hyperparam tuning
- [ ] Testing
[ ] Keep options in the config file
[ ] Documentation

Idea:

I generated some slurm run code in another project before like this: https://github.com/vitusbenson/noisecascades/blob/main/noisecascades/host.py#L296

Culprits:

PyTorch Lightning has its own SLURM functionality, which may or may not interfere with any proposed development