jupyter-server / enterprise_gateway

A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
https://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Other
616 stars 223 forks source link

Enhance kernelspec installation user experience #654

Open lresende opened 5 years ago

lresende commented 5 years ago

Today we ask users to manually download and handle the installation of Jupyter Enterprise Gateway sample kernelspecs. They are also responsible to modify/update parameters on all installed kernelspecs (e.g. spark_home, etc)

wget https://github.com/jupyter/enterprise_gateway/releases/download/v2.0.0.dev2/jupyter_enterprise_gateway_kernelspecs-2.0.0.dev2.tar.gz

KERNELS_FOLDER=/usr/local/share/jupyter/kernels

tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.0.0.dev2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_cluster/ spark_scala_yarn_cluster/

tar -zxvf jupyter_enterprise_gateway_kernelspecs-2.0.0.dev2.tar.gz --strip 1 --directory $KERNELS_FOLDER/spark_scala_yarn_client/ spark_scala_yarn_client/

A better user experience would be to handle some of these internally, and enable users to just install a given set of kernelspecs, based on the platform being used, and with the ability to update specific configurations. Something like:

jupyter enterprise_gateway install --platform=yarn --spark_home=/usr/local/bin/apache-spark/

The Apache Toree kernel has a similar user experience, and we could use that implementation as an example.

kevin-bates commented 5 years ago

I think we're going to need to move the etc directory under enterprise_gateway in the repo so that the necessary files to assemble kernelspecs will be available following pip install.

I also believe this work should happen before #655 since it may circumvent that need entirely as values could be made available via the configuration file. The benefit of jinja templates, otoh, is that they would provide access to the "body" of the kernel.json file. However, I think we might find that that body is essentially empty since it would almost be entirely derived from configuration values.