Use profileSpawner with configuration provided by z2jh

Ph0tonic commented 10 months ago

Bug description

Hi, I would like to use this ProfilesSpawner with https://github.com/jupyterhub/zero-to-jupyterhub-k8s. z2jh comes with an handy configuration to spawn pods in Kubernetes.

Is there a way to forward the configuration to a profile of this spawner ?

How to reproduce

My idea was to forward the config of KubeSpawner directly to the ProfilesSpawner however I get an error. Here is some of the config that I tried and would love to work.

c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'

c.ProfilesSpawner.profiles = [
    ('Kube', 'singleuser', 'kubespawner.KubeSpawner', c.KubeSpawner),
    # Some other spawner
]

And here is the error :

[I 2024-01-03 13:52:46.538 JupyterHub provider:659] Creating oauth client jupyterhub-user-my_user
                'tolerations': [{'effect': 'NoSchedule',                                                                                                                                                                                                                                                                    
                               'key': 'hub.jupyter.org/dedicated', 
                               'operator': 'Equal', 
                               'toleration_seconds': None,
                               'value': 'user'}, 
                              {'effect': 'NoSchedule', 
                               'key': 'hub.jupyter.org_dedicated', 
                               'operator': 'Equal', 
                               'toleration_seconds': None,
                               'value': 'user'}], 
              'topology_spread_constraints': None,
              'volumes': [{'aws_elastic_block_store': None,
                           'azure_disk': None,
                           'azure_file': None,
                           'cephfs': None,
                           'cinder': None,
                           'config_map': None,
                           'csi': None,
                           'downward_api': None,
                           'empty_dir': None,
                           'ephemeral': None,
                           'fc': None,
                           'flex_volume': None,
                           'flocker': None,
                           'gce_persistent_disk': None,
                           'git_repo': None,
                           'glusterfs': None,
                           'host_path': None,
                           'iscsi': None,
                           'name': 'volume-my-user', 
                           'nfs': None,
                           'persistent_volume_claim': {'claimName': 'claim-my-user'}, 
                           'photon_persistent_disk': None,
                           'portworx_volume': None,
                           'projected': None,
                           'quobyte': None,
                           'rbd': None,
                           'scale_io': None,
                           'secret': None,
                           'storageos': None,
                           'vsphere_volume': None}]},
     'status': None}
    Traceback (most recent call last): 
      File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2482, in _make_create_pod_request 
        await asyncio.wait_for(
      File "/usr/local/lib/python3.11/asyncio/tasks.py", line 479, in wait_for 
        return fut.result() 
               ^^^^^^^^^^^^ 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api 
        raise e 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api 
        response_data = await self.request(
                        ^^^^^^^^^^^^^^^^^^^ 
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 230, in POST 
        return (await self.request("POST", url,                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                                                                                                                                           
      File "/usr/local/lib/python3.11/site-packages/kubernetes_asyncio/client/rest.py", line 187, in request 
        raise ApiException(http_resp=r)
    kubernetes_asyncio.client.exceptions.ApiException: (422)
    Reason: error 
    HTTP response headers: <CIMultiDictProxy('Audit-Id': '29fe0728-ee34-4464-abcb-383eb2704e94', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': '8d66a213-297f-44ef-9dd7-f701c05a33e7', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'e09e092b-d2b8-4aa1-913d-7f6d6bbda951', 'Date': 'Wed, 03 Jan 2024 13:52:46 GMT', 'Content-Length': '397')>                                                                                                                                                                                                                                                
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod \"jupyter-my-user\" is invalid: spec.containers[0].ports[0].containerPort: Required value","reason":"Invalid","details":{"name":"jupyter-my-user","kind":"Pod","causes":[{"reason":"FieldValueRequired","message":"Required value","field":"spec.containers[0].ports[0].containerPort"}]},"code":422}

I am not very familiar with traitlets and how they work. I guess that I am missing something there. From what I understood, some default configurations are not loaded this way. Any help would be welcome, it could be great to be able to combine easily z2jh and this project.

\cc @consideRatio

Expected behaviour

I would expect to pass a valid configuration.

Actual behaviour

It currently fails.

welcome[bot] commented 10 months ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

consideRatio commented 10 months ago

I've never learned how ProfileSpawner works, but I see that you configure it to recognize it wraps KubeSpawner, and I assume you have the hub pod's image include this project to get this far.

I'm unable to allocate time to dig into the details about this, but here are some relevant debugging notes:

A z2jh deployed jupyterhub has a jupyterhub_config.py file that considers various configuration passed via helm chart values, this file is seen at https://github.com/jupyterhub/zero-to-jupyterhub-k8s/tree/main/jupyterhub/files/hub --- is everything done there compatible?
The error you see indicates KubeSpawner is active in generating a pod manifest, but it apparently isn't a valid one because spec.containers[0].ports[0].containerPort is reported required and missing. What code in kubespawner sets that, and why isn't that set etc? Note that if this was an error reported by itself, I'd suspect it was caused by some config, maybe via extra_container_config or similar making that part become overridden.

Ph0tonic commented 10 months ago

Thank you very much for your feedback! I noticed, I was not using the latest version of KubeSpawner and after the update I got another issue. See https://github.com/jupyterhub/wrapspawner/pull/59 if you are interested.

I will now try to fix this issue with the help of your debugging notes :+1: and post any progress here.

Ph0tonic commented 10 months ago

All right, so after some research, I was finally able to identify all variables that were overridden by WrapSpawner. Here is the config that I used to make it work with the base config of z2jh:

c.ProfilesSpawner.profiles = [
    ('Kube', 'kubespawner', 'kubespawner.KubeSpawner', {
        'ip':'0.0.0.0', 'port': 0, 'cmd': None, 'path': None, 'environment': {}, 'env_keep': []}),
    # Other spawner here
]

Without these variables overridden manually, the spawner is not able to launch any pod and connect to it.

Note: See https://github.com/jupyterhub/wrapspawner/pull/59 and https://github.com/jupyterhub/wrapspawner/pull/60

jupyterhub / wrapspawner