search

jupyterhub / zero-to-jupyterhub-k8s

Helm Chart & Documentation for deploying JupyterHub on Kubernetes
https://zero-to-jupyterhub.readthedocs.io
Other
1.51k stars 789 forks source link

Spawn failed: Expected object of type 'dict' (or 'V1Container') but got 'V1Container'. #2677

Closed orboan closed 2 years ago

orboan commented 2 years ago

Bug description

After docker image is built using:

./zero-to-jupyterhub-k8s/images/hub/Dockerfile

from the latest commit in the main branch.

And after deploying it using next config.yaml:

singleuser:
.....
hub:
  image:
    name: <image built with the Dockerfile found in the repo>
    tag: x.x.x
  config:
    extraEnv:
      LTI_AUTH_CONSUMER_URL: 'https://mydomain.com'
    security: |
      from kubernetes import client
      def modify_pod_hook(spawner, pod):
        pod.spec.containers[0].security_context = client.V1SecurityContext(
            privileged=True,
            capabilities=client.V1Capabilities(
                add=['SYS_ADMIN']
            )
        )
        return pod
      c.KubeSpawner.modify_pod_hook = modify_pod_hook
      c.JupyterHub.admin_access = True

Jupyterhub is loading with no problem (I needed to add pip install kubernetes to the Dockerfile, as it is a missing required library), but when spawning the jupyterlab image, I get:

Spawn failed: Expected object of type 'dict' (or 'V1Container') but got 'V1Container'.

The command used for deployment is:

#!/bin/bash
RELEASE=jupyterhub
NAMESPACE=hub
VERSION=1.2.0

helm upgrade --cleanup-on-fail \
  --install $RELEASE jupyterhub/jupyterhub \
  --namespace $NAMESPACE \
  --create-namespace \
  --version=$VERSION \
  --values config.yaml

Expected behaviour

The jupyterlab image spawns properly.

Do not know what could be the cause. This has started to happen after I built a new docker image of jupyterhub from the latest Dockerfile found in the main branch of the repo. And after upgrading from version 1.1.3 to 1.2.0 of the helm chart.

Actual behaviour

Does not spawn the jupyterlab image. It shows the error:

Spawn failed: Expected object of type 'dict' (or 'V1Container') but got 'V1Container'.

The hub pod logs are next:

[I 2022-04-29 14:46:07.876 JupyterHub provider:607] Creating oauth client jupyterhub-user-jovyan
[I 2022-04-29 14:46:07.899 JupyterHub spawner:2351] Attempting to create pvc claim-jovyan, with timeout 3
[I 2022-04-29 14:46:07.900 JupyterHub log:189] 302 POST /hub/spawn -> /hub/spawn-pending/jovyan (jovyan@::ffff:172.17.0.1) 64.51ms
[I 2022-04-29 14:46:07.908 JupyterHub pages:401] jovyan is pending spawn
[I 2022-04-29 14:46:07.912 JupyterHub log:189] 200 GET /hub/spawn-pending/jovyan (jovyan@::ffff:172.17.0.1) 6.76ms
[I 2022-04-29 14:46:07.913 JupyterHub spawner:2367] PVC claim-jovyan already exists, so did not create new pvc.
[E 2022-04-29 14:46:07.921 JupyterHub user:815] Unhandled error starting jovyan's server: Expected object of type 'dict' (or 'V1Container') but got 'V1Container'.
[E 2022-04-29 14:46:07.951 JupyterHub gen:623] Exception in Future <Task finished name='Task-346' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:945> exception=AttributeError("Expected object of type 'dict' (or 'V1Container') but got 'V1Container'.")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 952, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 833, in spawn
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 732, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 2499, in _start
        pod = await self.get_pod_manifest()
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/spawner.py", line 1875, in get_pod_manifest
        return make_pod(
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/objects.py", line 578, in make_pod
        pod.spec.init_containers = [
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/objects.py", line 579, in <listcomp>
        get_k8s_model(V1Container, obj) for obj in init_containers
      File "/usr/local/lib/python3.8/dist-packages/kubespawner/utils.py", line 98, in get_k8s_model
        raise AttributeError(
    AttributeError: Expected object of type 'dict' (or 'V1Container') but got 'V1Container'.

[I 2022-04-29 14:46:07.999 JupyterHub log:189] 200 GET /hub/api/users/jovyan/server/progress (jovyan@::ffff:172.17.0.1) 4.24ms
[I 2022-04-29 14:51:24.367 JupyterHub log:189] 200 GET /hub/api/ (cull-idle@127.0.0.1) 15.38ms

How to reproduce

Follow my explanation above.

Your personal set up

This is a deployment of jupyterhub in a on-premises kubernetes cluster, built using kubeadm.

Full environment ``` # paste output of `pip freeze` or `conda list` here ```
Configuration ```python # jupyterhub_config.py ```
Logs ``` # paste relevant logs here, if any ```
welcome[bot] commented 2 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

consideRatio commented 2 years ago

The unreleased version has switched to kubernetes_asyncio client from kubernetes client. You may have your code function by simply replacing kubernetes with kubernetes_asyncio in the import.

IMPORTANT: If you are using the helm chart version 1.2.0, you should use the docker image for version 1.2.0 as a base, not building from the main branch's Dockerfile. If you are using the latest dev release of the Helm chart, you should use its configured docker image as a base image etc! Make sure you don't try use a hub image that doesn't match the helm chart's version.