vmware / versatile-data-kit

One framework to develop, deploy and operate data workflows with Python and SQL.
Apache License 2.0
424 stars 56 forks source link

Standardise Additional Steps in Notebook Deployment Process #2899

Open duyguHsnHsn opened 10 months ago

duyguHsnHsn commented 10 months ago

What is the feature request? What problem does it solve? Currently, deploying a notebook job requires additional steps that are only integrated into the vdk-jupyterlab-extension, which is primarily for the Jupyter UI. This approach limits the standardisation of the deployment process, as these additional steps are not automatically executed when deploying from environments other than the Jupyter UI. This inconsistency poses challenges when using notebooks in different contexts, such as deploying from the command line interface (CLI) or other notebook interfaces. The need is for a standardised approach that ensures these additional steps are uniformly executed regardless of the deployment environment.

Suggested solution This should be an integral part of the deployment process, regardless of whether the deployment is done through the Jupyter UI or other interfaces such as the CLI. The best place it would fit is the vdk-notebook plugin. The solution should automatically include these steps in every deployment scenario, ensuring consistency and reliability across different environments. Additionally, this standardisation should extend to all operations that might require these additional steps, not just deployment, to maintain uniformity in the workflow. If the changes are done vdk-jupyterlab-extension should then use the new module to run those additional steps and the logic itself should be removed from the plugin.

duyguHsnHsn commented 10 months ago

Create Job operation is similar and should be standardised, as well.