Regarding c.KubeSpawner.environment and the EnvVar structure's valueFrom field

[qzchenwl]

I tried

hub:
  extraConfig:
    jupyterlab: |
      c.KubeSpawner.extra_container_config = {
        "env": [{
          "name": "MY_POD_IP",
          "valueFrom": {
            "fieldRef": {
              "fieldPath": "status.podIP"
            }
          }
        }]
      }

But this will overwrite the default env, and cause error. Any way to append env instead of replace all env?

[zlanyi]

what you think about in deployment,yaml of your helm chart?

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jupyterhub
spec:
   ...
  template:
    ...
    spec:
      ...
      containers:
        ...
        env:
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP

[qzchenwl]

@zlanyi This way would inject hub's POD_IP to hub container. What I need is for singleuser container.

[zlanyi]

@qzchenwl no, it's only declaration. What do you want to do? Do you want to have the IP of Hub on the Notebook server?

[qzchenwl]

@zlanyi I want the notebook server to be able to tell others how to connect to it. (make zmq listen on public IP instead of 127.0.0.1)

[abinet]

I have same problem. Trying to setup jupyter notebook with spark connecting to yarn cluster. For this i need SPARK_LOCAL_IP to be set in the notebook pod. Currently it seems there is no way to add environment variable with "valueFrom" instead of "value".

[consideRatio]

I think the key issue is that you want to write env in a more complicated form than just key:value where these are strings, we should perhaps try to support something like this though.

Hmmm oh wait okay now i was thinking of the helm chart config values under singleuser.env, but hmmm perhaps you can do this from the config anyhow?

Hmmmms eh it would be a technical ugly workaround... hmmmm... mindwrangling... kubespawner has a env field, right?

can you do something like you do above, but use the extend function instead on c.KubeSpawner.environment ?

[consideRatio]

c.KubeSpawner.environment.extend({
          "name": "MY_POD_IP",
          "valueFrom": {
            "fieldRef": {
              "fieldPath": "status.podIP"
            }
          }
        })

I did this from my mobile and memory, please fix typo/indentation etc

[abinet]

Thanks @consideRatio but this does not work for me. I'm getting the error message that Dict does not have "extend" method

[consideRatio]

@abinet ah, so c.KubeSpawner.environment it is a dictionary rather than an array... and that is a limitation because kubespawner have assumed all environment varibles will be hardcoded strings...

Okay, I think this will require kubespawner to adjust.

Then, until kubespawner is remade, the ugly option I then think may be required is something like this:

hub:
  extraConfig:
    jupyterlab: |
      env = [{
          "name": "MY_POD_IP",
          "valueFrom": {
            "fieldRef": {
              "fieldPath": "status.podIP"
            }
          }
        }]
      env.extend({'name': key, 'value': value} for key, value in c.KubeSpawner.environment.items())
      c.KubeSpawner.extra_container_config = {
        "env": env
      }

# this was returned when i tried this locally
[
    {
        'name': 'MY_POD_IP',
        'valueFrom': {
            'fieldRef': {'fieldPath': 'status.podIP'}
        }
    },
    {'name': 'TEST_ENV', 'value': 'TEST_ENV_VALUE'}
]

[abinet]

@consideRatio we are almost there. But c.KubeSpawner.environment does not deliver the env variables created by Spawner and they will be overridden later. I'm not very familiar with python. How can I get run KubeSpawner.get_env() properly?

[consideRatio]

@abinet oh, hmmm, oh? If that is the case, perhaps you can run this logic in a pre_spawn_hook then?

def my_hook(spawner):
    # spawner is refers to the specific users KubeSpawner instance (one is created per user)
    # run the logic here...

c.KubeSpawner.pre_spawn_hook = my_hook

[abinet]

Thank you @consideRatio! Last point was really the hit.

Here is my solution for now:

hub:
  extraConfig:
    ipaddress: |
      from kubernetes import client

      def modify_pod_hook(spawner, pod):
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_POD_IP", None, client.V1EnvVarSource(None, client.V1ObjectFieldSelector(None, "status.podIP"))))
          return pod
      c.KubeSpawner.modify_pod_hook = modify_pod_hook

[consideRatio]

Wieee thanks for sharing this solution!!

[TomAugspurger]

FWIW, I think that something like https://github.com/jupyterhub/kubespawner/pull/461 adding extra_env would be pretty useful. In my case, I have my jupyterhub helm config settings some environment variables:

jupyterhub:
  singleuser:
    extraEnv:
        DASK_GATEWAY__CLUSTER__OPTIONS__IMAGE: '{JUPYTER_IMAGE_SPEC}'
...

that I want to apply to all profiles. Then I have some profiles (e.g. with GPUs) that should have some extra environment variables. It'd be nice to have

...
    profileList:
      - display_name: gpu
        kubespawner_override:
          extra_env: {'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility'}

to append (or update / overwrite) those extra environment variables, while keeping the common ones. Would something like a KubeSpawner.extra_env be in scope?

[yuvipanda]

@TomAugspurger Wouldn't just setting environment act the same way? Since it's a dictionary, it'll just be updated.

[TomAugspurger]

Oh, I might have been mistaken in how it behaves. I probably assumed without testing that it would be replaced completly, rather than updated.

[consideRatio]

Hmmm kubespawner overrides will indeed override the configuration field i think.

To adjust config based on profile, i suggest hooks. I dont think it will be viable for maintenance and documentation to add more and more dynamics to manage the same config in kubespawner, like env and extra_env and extra_extra_env or similar.

At least if its not made in a config generic way.

[consideRatio]

Im on mobile, will probive a concrete example.

[consideRatio]

This can be set in hub.extraConfig for example.

async def pre_spawn_hook(spawner):
    auth_state = await spawner.user.get_auth_state()
    user_details = auth_state["oauth_user"]
    # 
    spawner.environment.update(
        {
            "USER_SOMETHING": user_details.get(user_details["something"], ""),
        }
    )

c.KubeSpawner.pre_spawn_hook = pre_spawn_hook

[TomAugspurger]

Thanks! I'll try that out.

[1ambda]

In case of resources, you can use V1ResourceFieldSelector for example,

      from kubernetes import client

      def modify_pod_hook(spawner, pod):
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_POD_IP", None, client.V1EnvVarSource(field_ref = client.V1ObjectFieldSelector(field_path = "status.podIP"))))
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_POD_NAME", None, client.V1EnvVarSource(field_ref = client.V1ObjectFieldSelector(field_path = "metadata.name"))))
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_CPU_REQUEST", None, client.V1EnvVarSource(resource_field_ref = client.V1ResourceFieldSelector(resource = "requests.cpu"))))
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_CPU_LIMIT", None, client.V1EnvVarSource(resource_field_ref = client.V1ResourceFieldSelector(resource = "limits.cpu"))))
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_MEMORY_REQUEST", None, client.V1EnvVarSource(resource_field_ref = client.V1ResourceFieldSelector(resource = "requests.memory"))))
          pod.spec.containers[0].env.append(client.V1EnvVar("MY_MEMORY_LIMIT", None, client.V1EnvVarSource(resource_field_ref = client.V1ResourceFieldSelector(resource = "limits.memory"))))
          return pod

      c.KubeSpawner.modify_pod_hook = modify_pod_hook

[cointreauu]

I edited @abinet 's code slightly, and got Spawn failed: 'coroutine' object has no attribute 'metadata' error..

        import os
        async def modify_pod_hook(spawner, pod):
            output = os.popen("dbus-launch --sh-syntax").read().split('\n')
            pod.spec.containers[0].env.append("os_asdf", "os_asdf")

            return pod
        c.KubeSpawner.modify_pod_hook = modify_pod_hook

[consideRatio]

Renove the async keyword

[cointreauu]

@consideRatio thanks, it worked. It should be a dictionary format including name and value key.

        import subprocess
        def modify_pod_hook(spawner, pod):
            pod.spec.containers[0].env.append({"name": "os_asdf", "value": output})

            return pod
        c.KubeSpawner.modify_pod_hook = modify_pod_hook

[samyuh]

@cointreauu your solution is perfect, but you don't need to import subprocess module :)

def modify_pod_hook(spawner, pod):
    pod.spec.containers[0].env.append({"name": "os_asdf", "value": "envvalue"})

    return pod
c.KubeSpawner.modify_pod_hook = modify_pod_hook