Ability to use sub run to execute a Notebook definition. This is helpful for long-running Notebooks that aren't a Dataset nor a Model.
Use case: Running a Notebook that might take 10+ hours, for example, I wanted to filter out valid K8s YAML manifests from the-stack dataset which downloads 100+GB of data and in the notebook I push the result back to HuggingFace datasets.
Expected behavior:
If inside the directory there is a Notebook resource YAML file then also allow running that
The run or view command should show the HTML or rendered ipynb file and refresh that every 10 seconds
Why?
Provide more flexibility to folks that just want to run a notebook and watch the results remotely whenever they want to. The notebook command launches jupyter lab but once you exit the browser session you have no way of getting back the results of the run.
Alternatives
Utilize the Dataset or Model resource for Notebooks that aren't technically either
Re-design of how Substratus core resources and shared filesystem be namespace scoped instead of resource scoped
Ability to use
sub runto execute a Notebook definition. This is helpful for long-running Notebooks that aren't a Dataset nor a Model.Use case: Running a Notebook that might take 10+ hours, for example, I wanted to filter out valid K8s YAML manifests from
the-stackdataset which downloads 100+GB of data and in the notebook I push the result back to HuggingFace datasets.Expected behavior:
Why?
notebookcommand launches jupyter lab but once you exit the browser session you have no way of getting back the results of the run.Alternatives