Open SarahAlidoost opened 3 months ago
Good point @SarahAlidoost . I believe that the reason why the repository is currently copied to the remote (and not cloned) is that in this way you can modify the scripts locally before uploading them to the platform. But this allows you to modify these scripts only when installing, you will be bound to use these scripts for all time you will be running jupyter..
Maybe a better way to do this could be:
jupyter-dask-on-slurm
locally, you create a local folder (.config/jupyterdask-on-slurm
?) where you put all the job scripts (jupyter_dask_spider.bsh
, jupyter_dask_snellius.bsh
, etc.) and platform configurations file (platforms.ini
)jupyter-dask-on-slurm
to start jupyter on the remote, the correct job file is copied to the platform to a temporary folder (/tmp
?) and submitted. Credentials to access the platform would also be in a standard location, so we don't need to track where the platform.ini
file is. This way you can update the job script locally and this will be immediately reflected when starting jupyter. What do you think? When I am back I could try to implement these changes and open a PR towards fix_71
, so that #75 will then solve all these at once.
Hi @SarahAlidoost and @rogerkuou sorry but I could not get that far in what I indended to do on updating the script to work as I was trying to describe in the previous message. The discussion with @rogerkuou made me think that maybe we should not modify too much the way in which Dask configuration is provided, i.e. via config files, so maybe we should not implement passing parameters via environment variables (see third point above).
I still think we should avoid having to clone the repository on the remote - maybe we could even drop the installation part from the Python script? I have the impression we have been the only ones using it in the courses, while the other people that have been using RS-DAT have preferred to go through the manual installation.. Also, seeing how things have evolved for Spider, maybe we should just give more flexibility on which environment should be used and avoid pushing the use of conda. See also #82.
Maybe worth it to discuss it briefly together once we are all back?
Currently, in the function
install_JD
:while the function
copy_folder
copies a folder instead of cloning. So, the print statement needs a fix. Also, in the functioncopy_folder
, a relative path../JupyterDaskOnSLURM
is used which means first cloning the repo and cd to the directory. Why the whole folderJupyterDaskOnSLURM
on remote is needed? can we just copy the scripts that are needed?