Adding custom kernel to EG

[lucabem]

I am working with Jupyter Enterprise Gateway, but i am not understanding at all how EG works with Kubernetes. My question is related to how EG finds kernels such as r_kubernetes or python_kubernetes. I have deployed EG as a service inside my Kubernetes cluster

If I want to add a new kernel, where i should place it? As i have said before, because of treat EG as a Kubernetes Service, i cant find R's kernel.json but i am able to reproduce R on a Notebook.

In resume, how could i add a custom kernel? Should i create /etc/kernelspecs dir?

Hope I could explain my problem.

[kevin-bates]

Thanks for the question @lucabem.

Yes, this can be a little confusing due to the containerized nature of things. The kernelspec files are located in EG image under /usr/local/share/jupyter/kernels (just like the on-prem offering). In order to "bring your own spec", you'll need to expose the built-in "sample" specs. This can be done in different ways, two of which - NFS or a custom container image - are discussed here in the Kubernetes section of the docs.

If you'd also like to associate a different kernel image (pod) with this kernel, you'll also want to read Custom Kernel Images - which also talks about kernelspec modification here as well.

[lucabem]

Thanks for responding @kevin-bates, now i have everything clearer. It seems like i was trying to connect my NFS-server with the worker node, but i should connect server with JEG-POD, shouldnt i?

Now, I would place kernels on NFS-Server and EG will list them by default. Tomorrow I will try it and keep you informed. Thanks!

[lucabem]

Hi @kevin-bates! Finally I was able to configure NFS-Server out of the cluster. I leave the code here, it might help someone (im working on Centos 7)

yum install nfs-utils systemctl enable rpcbind && systemctl enable nfs-server && systemctl enable nfs-lock && systemctl enable nfs-idmap

systemctl start rpcbind && systemctl start nfs-server && systemctl start nfs-lock && systemctl start nfs-idmap

vim /etc/exports We add that line -> /usr/local/share/jupyter/kernels *(rw,sync,no_root_squash,no_all_squash)

systemctl restart nfs-server

firewall-cmd --permanent --zone=public --add-service=nfs firewall-cmd --permanent --zone=public --add-service=mountd firewall-cmd --permanent --zone=public --add-service=rpc-bind firewall-cmd --reload

[kevin-bates]

Thank you for the NFS reference. Please close the issue once you feel it has been resolved. Thanks.

[lucabem]

Hi @kevin-bates Is it possible to let user creates his own enviroment and use it inside Kubernetes Cluster?

1) User creates his own env with his libraries. 2) He logs in JupyterHub and on the Kernels's combobox appears a kernel with his env (Python_Modified) 3) JH spawns a notebook on cluster where user needn't to install libraries because they have been installed on his env

[kevin-bates]

Hi @lucabem - I'm not sure I'm familiar enough with the JH configuration to answer your question.

EG spawns kernel pods, each of which can be customized in a generic manner since the corresponding image is used by different users. However, we also "flow" any KERNEL_-prefixed environment variables from Notebook thru EG to the kernel invocation, so per-user tuning could occur via that manner, where the kernel-pod.yaml jinja template can reference those per-user variables.

You would then need to probably do something similar in the configuration between JH and the Notebook pod or derive those per-user KERNEL_ env variables in some manner.

Not sure this helps, nor am I sure how you'd do this with just JH and Notebook (i.e., w/o EG and kernel-specific pods), but that could be another option for you.

[lucabem]

Yes, i am thinking on pass the param thru EG.

Im a bit confused on the way proccess knows which kernel should start notebook. I mean, when you click on the kernel you want to spawn on the combobox there could be some iteration where you said "Spawn Python with TF", because if it occurs maybe i could send a parameter.

I have seen on RoadMap that you would add user enviroments. I think the main problem is how to pass parameter to kernel. I have think that if we are able to pass that param we could modify kernel-py adding "conda activate $param" and make a soft-link on NFS with user-env

[kevin-bates]

If by "kernel you want to spawn on the combobox", are you referring to the drop-down list in the notebook itself? or some combobox presented by JupyterHub before the Notebook server is spawned?

EG only deals with the kernels available from within a given Notebook. The Notebook software obtains the list of kernels via the /api/kernelspecs REST call - which, when Notebook is configured to use EG, comes from the EG server. That list is derived from the kernels configured on the EG server.

JH probably allows selection of various Notebook images where the image is essentially configured for the underlying kernel it's going to spawn directly (and within its container -rather than EG spawning the kernel across a managed cluster).

There is a lot of momentum on what we're calling parameterized kernels and I believe that's what you're driving at as well. The idea I'm proposing here is that the kernelspec would provide JSON schema describing what parameters are available to a given kernel and, in some cases, their set of values, etc. Some parameters may be used to configure the kernel's environment (e.g., GPU access, memory, CPUs, etc.) while others will be conveyed directly to the kernel, similar to how {connection_file} works today. This infrastructure is not currently in place as it needs to be present across the stack.

[lucabem]

If by "kernel you want to spawn on the combobox", are you referring to the drop-down list in the notebook itself?

Yes, i was confused with the work that JEG does.

I dont know if what i am trying to integrate is impossible. My purpose is letting user to create their own enviroments with nb_conda_kernels and use a generic python image which activates conda eviroment specified via param

For example

# we have modifed conda to install envs on NFS --> /user/local/share/jupyter/kernels
conda create -n myEnv && install libraries_user_want

#on Kernel.json
image_name: generic_python_image

Now JEG will provide kernel Python with Envs to Notebook Server. When user clicks on drop-down 'Python with Envs' options, we create pod with generic_image which has a line conda activate $PARAM.

Finally user could use his own environment.

I think if we are able to share environments we could reach it. Hope I have explained it as well as possible.

[kevin-bates]

Thank you for your explanation - that is helpful. So if you have the ability to launch the kernel within an environment (I suspect this could be done via a "run.sh" similar to what many of the EG kernelspecs do - or convey the env to launch to the container and have its boot script do the right thing) then its really a matter of conveying the parameter.

Unfortunately. there's no "baked in" facility for prompting for such a parameter (or knowing what values are appropriate) at this time. Prior to having parameterized kernel support, you'd probably need to build your own notebook extension that, when invoked, presents the dialog to prompt, at which time, you could place a "KERNEL_CONDA_ENV" variable into the environment - which the gateway package in NB would recognize and add it to the payload for the start kernel request.

If you were to name various conda envs on the user's name - or some algorithm with the user's name as the base (e.g., luis_env), then you could leverage KERNEL_USERNAME for the "parameter" portion of the ultimate target and not require an extension.

If you're front end is using the REST api directly to start and manage kernels, then you're more free to not have to set an env variable, but can just manipulate the env stanza in the kernel start request's json body directly.

I hope that helps.

[lucabem]

Thank you @kevin-bates - that is helpful.

this could be done via a "run.sh" similar to what many of the EG kernelspecs do

What do you mean with that?I think before kernel-py would call bootstrap-kernel.sh (or before start_ipython) we could activate de environment, which is named USER_env. This environment could be located on the NFS-Server and conda could be able to find it.

just manipulate the env stanza in the kernel start request's json body directly.

On kernel'json file there is a field env but I dont understand what you are trying to say.

[kevin-bates]

Sorry for the run.sh comment. On Spark envs (including k8s) we launch the kernels using a run.sh script. However, you're using the vanilla kernels, so the kernels are launched using the launch_kubernetes.py script - I wasn't making that connection. This script is responsible for gathering parameters (typically env variables) and ensuring they are reflected in the kernel-pod.yaml file EG uses to launch the kernel pod. You must first get the parameters conveyed into the container before the container can do anything with them.

Then, in the container, you would have the bootstrap-kernel.sh "act on" the variables - either directly or indirectly via a script it launches.

The env stanza in the kernel.json are environment variables that the launch framework ensures are available to the kernel. Because these are created at deployment time, they really can't include information that varies from request to request. However, if these remain constant across users, you could configure different kernel.json files (in different directories) for each variance in the env stanza. One could specify that one conda env be used, while the other indicates another. However, I get the impression you want more variability than that, but thought I'd mention it.

For env information that varies from request to request (i.e., one kernel's startup to another's), you'd need to get those values into the actual REST payload for the start kernel request. Since the NB gateway package (or, if you're using NB < 6.0 the NB2KG extension) automatically propagates into the payload all envs prefixed with KERNEL_, then you'd set those in your user's environment.

This all predates parameterized kernels - which we hope to have available at some point (but probably on the order of multiple months since so many layers require supporting changes). Once "PK"s are available, then parameters are truly dynamic per request (and the number of kernel.json files will decrease significantly).

[lucabem]

Thank you @kevin-bates - I guess it will work. I would try to modify bootstrap-kernel-sh to activate the conda environment passed thru KERNEL_ENV on kernel.json.

With that solution, at least we have just one docker image. I keep you informed.

[kevin-bates]

My guess is that its still owned by root and you won't have access to it as users jovyan. Also, when I pulled your tar file, bootstrap-kernel.sh's launch_python_kernel() didn't contain the KERNEL_ENV variant. It only had this...

launch_python_kernel() {
    # Launch the python kernel launcher - which embeds the IPython kernel
and listens for interrupts
    # and shutdown requests from Enterprise Gateway.

    export JPY_PARENT_PID=$$  # Force reset of parent pid since we're
detached

set -x
/usr/local/share/jupyter/kernels/entornoCompartido/bin/python
${KERNEL_LAUNCHERS_DIR}/python/scripts/launch_ipykernel.py
--RemoteProcessProxy.kernel-id ${KERNEL_ID}
--RemoteProcessProxy.response-address ${EG_RESPONSE_ADDRESS}
--RemoteProcessProxy.spark-context-initialization-mode
${KERNEL_SPARK_CONTEXT_INIT_MODE}
{ set +x; } 2>/dev/null
}

Btw, if you find this working, I think we'd be open to having you provide a PR for KERNEL_ENV support. This seems like it might be useful. So a single image would support multiple envs (and thus different packages) rather than having multiple images?

On Thu, Nov 7, 2019 at 3:40 AM Luis Cabezon Manchado < notifications@github.com> wrote:

Hi @kevin-bates https://github.com/kevin-bates!

I have built my own Dockerfile. Its the same kernel-py dockerfile but instead of dowloading your 2.0.0 release, i havre create my onw release.

I have modified bootstrap-kernel.sh to activate my env.

launch_python_kernel() { export JPY_PARENT_PID=$$ # Force reset of parent pid since we're detached

set -x
/usr/local/share/jupyter/kernels/${KERNEL_ENV}/bin/python ${KERNEL_LAUNCHERS_DIR}/python/scripts/launch_ipykernel.py --RemoteProcessProxy.kernel-id ${KERNEL_ID} --RemoteProcessProxy.response-address ${EG_RESPONSE_ADDRESS} --RemoteProcessProxy.spark-context-initialization-mode ${KERNEL_SPARK_CONTEXT_INIT_MODE}
{ set +x; } 2>/dev/null

}

And Dockerfile is (following doc):

FROM jupyter/scipy-notebook:61d8aaedaeaf USER root

RUN wget https://github.com/lucabem/enterprise_gateway/releases/download/v2.0.1/jupyter_enterprise_gateway_kernel_image_files-2.0.1.tar.gz &&\ tar -xvf jupyter_enterprise_gateway_kernel_image_files-2.0.1.tar.gz -C /usr/local/bin &&\ rm -f jupyter_enterprise_gateway_kernel_image_files-2.0.1.tar.gz &&\ fix-permissions /usr/local/bin

USER $NB_UID ENV KERNEL_LANGUAGE python CMD /usr/local/bin/bootstrap-kernel.sh

When I use image im getting an error /bin/bash 1: /usr/local/bin/bootstrap-kernel.sh not found. If I just change the wget's url to your release it works.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jupyter/enterprise_gateway/issues/747?email_source=notifications&email_token=AFMNPCFVJZAZZ476W5LRDKTQSP5BZA5CNFSM4JDM4G42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDMEGIY#issuecomment-551043875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFMNPCDGYKFNQFQL6BCN46LQSP5BZANCNFSM4JDM4G4Q .

[kevin-bates]

@lucabem - I think you're also going to want to update the kernel-pod.yaml template to handle KERNEL_ENV. Since this might not always be used, having conditional logic like that for working_dir (but in the env section) would be wise.

((We should extend the template to walk the set of keywords whose names are prefixed with kernel_ (lower case) and build the env set dynamically (relative to KERNEL_).))

[lucabem]

Hi @kevin-bates!

Yes, we have modified kernel-pod.yaml to allow Kernel_ENV. In addition, we have modified it to allow pod finds environment from NFS (same code as enterprise.yaml) .

Also, when I pulled your tar file, bootstrap-kernel.sh's launch_python_kernel() didn't contain the KERNEL_ENV variant. It only had this...

I have been worked on locally. What you see is the path to an env created by me, it was just a test. Furthermore, my kenel_images_files.tar.gz was zipped wrong. I mean, instead of using tar -cvf ., I was using tar -cvf bootstrap-kernel.sh kenerl-launchers, so it was packing i a worng way.

I have built sucessfully the kernel-py image but i didnt do more. Tomorrow i would try it and i write you.

[lucabem]

Hi @kevin-bates. This issue could be closed. If we create an NFS where environments are installed and we share that NFS with pods, it is possible to use one generic image with different conda envs.

We have reached it. On bootstrap-kernel.sh we modify python ${KERNEL_LAUNCHERS_DIR} ... to path/to/env/bin/python {KERNEL_LAUNCHERS_DIR}....

Thanks for your help!!

[kevin-bates]

Sounds good. I think a nice addition would be to use an env variable, like KERNEL_PYTHON, that is set in the kernel.json's env: stanza and whose value is specific to the path within the docker image that is also referenced in the kernel.json file. Something like this...

{
  "language": "python",
  "display_name": "My Custom KernelSpec",
  "metadata": {
    "process_proxy": {
      "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessProxy",
      "config": {
        "image_name": "my custom image that uses entornoCompartido"
      }
    }
  },
  "env": {
     "KERNEL_PYTHON": "/usr/local/share/jupyter/kernels/entornoCompartido/bin/python"
  },
...

Then, modify bootstrap-kernel.sh to use ${KERNEL_PYTHON} ${KERNEL_LAUNCHERS_DIR} ... with the default setup in bootstrap-kernel.sh as:

KERNEL_PYTHON=${KERNEL_PYTHON:-python}

so today's code just works.

Note that since we don't currently have the ability to automatically add all KERNEL_ env variables in the kernel-pod.yaml template, you'd likely need to modify the template to also include the conveyance of KERNEL_PYTHON to the launch - otherwise the default of python will be used.

At any rate, please close this issue once you're satisfied our discussion is complete. Thank you for using Enterprise Gateway.