Feature Suggestion: Remote cluster submitit launcher - Githubissues

mila-iqia / ResearchTemplate

Research Project Template Repository

5 stars 1 forks source link

Feature Suggestion: Remote cluster submitit launcher #66

Open lebrice opened 6 days ago

lebrice commented 6 days ago

Idea

You have the repo checked out on your local machine.
You want to submit a job on the Mila cluster. You have SSH access to the Mila cluster.
You launch a multirun, with something like cluster=mila, and your jobs get executed on the Mila cluster remotely.

How?

create a custom launcher plugin for Hydra, similar to the existing submitit launcher plugin.
This plugin can probably reuse the convenience utilies from milatools, in particular the RemoteV2, salloc, sbatch and ComputeNode classes.
Sync the project state on the cluster to match the current state.
Execute the job on the cluter (using the submitit launcher plugin)
Sync the results locally and display them (or any errors).

lebrice commented 6 days ago

@tesfaldet I think this might be of interest to you. Let me know if you have any ideas or thoughts about this :)