search

jupyter-server / jupyter-scheduler

Run Jupyter notebooks as jobs
https://jupyter-scheduler.readthedocs.io
BSD 3-Clause "New" or "Revised" License
187 stars 23 forks source link

Input file auto-update #524

Open hardcore-code opened 1 month ago

hardcore-code commented 1 month ago

Problem

When we create a new job definition, the notebook is copied somewhere in the server.

Quite often you change that file and the you need to recreate everything.

Proposed solution

It would be incredible to have the option to retain the link to the original file. Kind of NO_COPY. It would save a lot of time.

Would be nice to have a way to have the job connected to a file path instead of saving a copy of the file that requires a manual update...

It could be an option during the creation/editing.

The field "url" that you have in the db table could be used for that purpose....

Maybe you would need to add an additional flag field called "linked". Once it is true, scheduler uses the url as a link to the file. Once false, use as it is now, a copy to a specific folder in the server.

welcome[bot] commented 1 month ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

andrii-i commented 1 month ago

Hi @hardcore-code. Thank you for creating this issue. You can edit which input file / notebook job definition uses to create jobs using Edit Job Definition functionality. To do this:

  1. Go to the job definition detail page by clicking job definition name in the detail view Screenshot 2024-06-03 at 11 38 46 AM
  2. Choose "Edit Job Definition" Screenshot 2024-06-03 at 11 39 33 AM
  3. Drag a file from the filebrowser and drop it into the "Input file snapshot" input field. Then save changes Screenshot 2024-06-03 at 11 39 55 AM
andrii-i commented 1 month ago

@hardcore-code I've created an issue #525 to track necessity to document how to update input file snapshots. Please try the functionality as described above. Closing as solved by existing functionality but feel free to ask follow-up questions, provide additional feedback, or even reopen if my reply above does not solve your issue.

hardcore-code commented 1 month ago

Lightning fast response Andrii. That is a way to do it (I did not know). Thanks for that. It already helps.

I trust it does not eliminate the need for the linked version....

Would be nice to have a way to have the job connected to a file path instead of saving a copy of the file that requires a manual update...

It could be an option during the creation/editing.

The field "url" that you have in the db table could be used for that purpose....

Maybe you would need to add an additional flag field called "linked". Once it is true, scheduler uses the url as a link to the file. Once false, use as it is now, a copy to a specific folder in the server.

Hey, very nice to propose all of this. I am certainly underestimating the complexity. But I trust other may have similar needs..

By the way, great tool.

Sent from Outlook for Androidhttps://aka.ms/AAb9ysg

From: Andrii Ieroshenko @.> Sent: Monday, June 3, 2024 8:51:06 PM To: jupyter-server/jupyter-scheduler @.> Cc: hardcore-code @.>; Mention @.> Subject: Re: [jupyter-server/jupyter-scheduler] Mutable input files (Issue #524)

@hardcore-codehttps://github.com/hardcore-code I've created an issue #525https://github.com/jupyter-server/jupyter-scheduler/issues/525 to track necessity to document how to update input file snapshots. Please try the functionality as described above. Closing as solved by existing functionality but feel free to ask follow-up questions, provide additional feedback, or even reopen if my reply above does not solve your issue.

— Reply to this email directly, view it on GitHubhttps://github.com/jupyter-server/jupyter-scheduler/issues/524#issuecomment-2145899162, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMQ3JLLJS2QAZIXUSEX6EKTZFS3JVAVCNFSM6AAAAABIXA7TYSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBVHA4TSMJWGI. You are receiving this because you were mentioned.Message ID: @.***>

andrii-i commented 1 month ago

@hardcore-code

Would be nice to have a way to have the job connected to a file path instead of saving a copy of the file that requires a manual update...

It could be an option during the creation/editing.

The field "url" that you have in the db table could be used for that purpose....

Maybe you would need to add an additional flag field called "linked". Once it is true, scheduler uses the url as a link to the file. Once false, use as it is now, a copy to a specific folder in the server.

I think this makes a lot of sense as a feature request / potential new feature. Could you please edit the issue description adding the feature description quoted above to it? We could then reopen this issue and use it to track and discuss "Mutable input files" feature request. Otherwise no worries, I can create a new issue myself based on your comment.

By the way, great tool.

Than you, it's nice to hear that you find Jupyter Scheduler useful.

andrii-i commented 1 week ago

I've edited the name and description of this issue to use it as a feature request.