search

jupyter-server / jupyter-scheduler

Run Jupyter notebooks as jobs
https://jupyter-scheduler.readthedocs.io
BSD 3-Clause "New" or "Revised" License
193 stars 23 forks source link

Better files download and replication configurability #543

Open aaaditij opened 1 week ago

aaaditij commented 1 week ago

Problem

For the createjob api, one of the inputs to this API is a boolean flag called package_input_folder, which when set to true, packages the input folder (the folder containing the input notebook) and all nested files and subfolders within it during the job creation. This introduces the following problems:

  1. download_files api copies the entire input folder from staging area to the output folder. This is currently done so that notebook downloaded with other output files would have access to all the same files as original and so that running notebook as a whole or some cells could be replicated if they refer to files via local paths. This in essence is copying the entire input folder twice, once to the staging area and then to the output folder and can quickly lead to storage exhaustion if the input folder is large.

  2. The files in the staging area are never cleaned up again eating up storage space.

Proposed solution

  1. Add a boolean flag to download_files api to allow the user to specify if they only want the output files to be copied over to the output folder.
  2. Add a boolean flag to download_files api to delete all files belonging to an execution from the staging area after they have been copied over to the output folder.
andrii-i commented 1 week ago

Hi @aaaditij. Thank you for creating this enhancement request. These options make sense to me as traitlet-configurable options. At the same time we are not working on any of them now and are not planning to as there are some high-priority deliverables in the pipeline.