I propose we add a new module that we can use to easily connect to clusters and start jobs using Python.
Currently to run calculations using TCutility you need to first log in to a cluster (e.g. Bazis or Snellius) and then start your python scripts there. Often it would be nicer to be able to log in to the clusters from your local machine using Python. Let's say I am building a large database for a ML project and I have calculations stored on both Bazis and Snellius. To start the next batch of calculations I want to check if a calculation is already done on either cluster. Currently there is no easy way of checking that unless you do it manually. With the new connect module I can open a connection to both clusters, check if the calculations were done there, and then choose one of the clusters to run it on. We could then generate input files and runscripts locally and upload them to the cluster and submit them to Slurm.
I propose we add a new module that we can use to easily connect to clusters and start jobs using Python.
Currently to run calculations using TCutility you need to first log in to a cluster (e.g. Bazis or Snellius) and then start your python scripts there. Often it would be nicer to be able to log in to the clusters from your local machine using Python. Let's say I am building a large database for a ML project and I have calculations stored on both Bazis and Snellius. To start the next batch of calculations I want to check if a calculation is already done on either cluster. Currently there is no easy way of checking that unless you do it manually. With the new connect module I can open a connection to both clusters, check if the calculations were done there, and then choose one of the clusters to run it on. We could then generate input files and runscripts locally and upload them to the cluster and submit them to Slurm.