Pale-Blue-Dot-97 / Minerva

Minerva project includes the minerva package that aids in the fitting and testing of neural network models. Includes pre and post-processing of land cover data. Designed for use with torchgeo datasets.
MIT License
20 stars 1 forks source link

Use the `submitit_slurm` plugin with `hydra` to handle SLURM distributed computing #481

Closed Pale-Blue-Dot-97 closed 1 week ago

Pale-Blue-Dot-97 commented 4 months ago

Is your feature request related to a problem? Please describe. Switching to hydra has introduced issues with using multiprocessing on SLURM.

Describe the solution you'd like hydra provides a submitit plugin that can handle configuring SLURM jobs with hydra. This should fix the issue and simplify using SLURM with minerva. It is also a another step towards using hydra.sweep.

Describe alternatives you've considered None

Additional context None