A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across distributed clusters such as Apache Spark, Kubernetes and others.
We need to make sure kernels come up in a well-defined environment that is sufficiently isolated from that of other users. Considerations should include the user's home folder in HDFS or GPF and data persistence across kernel sessions.
Environments could be isolated on a ...
kernel by kernel basis
user by user basis
be "fresh" each time a kernel gets launched
For Python kernels the mechanism to use could be virtualenv or Anaconda.
For Java/Scala kernels, the submitted jar files constitute the runtime environment
lresende
commented
7 years ago
We need to make sure kernels come up in a well-defined environment that is sufficiently isolated from that of other users. Considerations should include the user's home folder in HDFS or GPF and data persistence across kernel sessions.
Environments could be isolated on a ...
For Python kernels the mechanism to use could be
virtualenvor Anaconda.For Java/Scala kernels, the submitted jar files constitute the runtime environment
For R kernels it could be Renv or Packrat.