search

AgnostiqHQ / covalent-slurm-plugin

Executor plugin interfacing Covalent with Slurm
https://covalent.xyz
Apache License 2.0
27 stars 6 forks source link

Allow for SLURM submission locally #55

Open Andrew-S-Rosen opened 1 year ago

Andrew-S-Rosen commented 1 year ago

What should we add?

The SLURM executor relies on SSHing into the machine and submitting the SLURM jobs. This is excellent, but it would be ideal if the user also had the option to use the SLURM executor locally if they are dispatching from the HPC machine itself. One could imagine this might be particularly relevant, for instance, if there are intense security measures that restrict SSH access in a way that the plugin can't work with (e.g. requiring a physical Yubikey).

To be clear, this would be a large feature addition, so I am not suggesting the maintainers or myself work on it ASAP, but I wanted to log this prior discussion here anyway in case someone feels ambitious. In the meantime, it should be possible to use the Dask executor to write SLURM jobs locally (perhaps worth a tutorial if it's ultimately too ambitious to include here).

Describe alternatives you've considered.

No response

santoshkumarradha commented 1 year ago

hey @arosen93 I think this is a neat feature.

IMO this does not seem like a lot of work, I think it should be as simple as abstracting https://github.com/AgnostiqHQ/covalent-slurm-plugin/blob/develop/covalent_slurm_plugin/slurm.py#L261C2-L266 asyncssh.connect , which returns a SSHClientConnection object and making it such that we have an local abstraction for this with a toggle switch inside the plugin kwargs.

Andrew-S-Rosen commented 1 year ago

@santoshkumarradha: Do you know if there is a way to use asyncssh such that the connection object returned simply performs actions on the local machine? I had trouble finding something like that in the docs. If there's an easy way to return a connection object that will run locally, I'll happily test things out and submit a PR.