search

jupyterhub / kubespawner

Kubernetes spawner for JupyterHub
https://jupyterhub-kubespawner.readthedocs.io
BSD 3-Clause "New" or "Revised" License
543 stars 304 forks source link

reflector:_list_and_update check k8s api response error before read the items #722

Open ealyn opened 1 year ago

ealyn commented 1 year ago

I got an KeyError: 'items' while using KubeIngressProxy, and I add some log to find out reason:

[I 2023-04-23 07:22:57.264 JupyterHub reflector:231] initial_resources keys: dict_keys(['kind', 'apiVersion', 'metadata', 'status', 'message', 'reason', 'details', 'code']) [I 2023-04-23 07:22:57.265 JupyterHub reflector:232] initial_resources: {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:jupyterhub:hub" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "jupyterhub"', 'reason': 'Forbidden', 'details': {'group': 'networking.k8s.io', 'kind': 'ingresses'}, 'code': 403} [E 2023-04-23 07:22:57.265 JupyterHub reflector:387] Initial list of ingresses failed Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 385, in start await self._list_and_update() File "/usr/local/lib/python3.9/site-packages/kubespawner/reflector.py", line 235, in _list_and_update for p in initial_resources["items"] KeyError: 'items'

Proposed change

  1. function reflector:_list_and_update check k8s api response error before read the items
  2. jupyterhub chart templates/hub/rbac.yaml add ingress api (for https://github.com/jupyterhub/helm-chart)
consideRatio commented 1 year ago

There may be another issue as well, but the key issue is permissions.

The jupyterhub chart isn't pre-configured to grant tge required permissions.

'Failure', 'message': 'ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:jupyterhub:hub" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "jupyterhub"'

I recall there are docstrings about required permissions to be found in the class defintion for KubeIngressProxy.

consideRatio commented 1 year ago

A fix for this project is to fail with a better error message if we receive a forbidden response.

dolfinus commented 1 year ago

reflector._list_and_update calls api.list_namespaced_ingress, but it does not raise exceptions if Kubernetes returned 403 error.

This is because _preload_content=False is used, and method parses response object manually: https://github.com/jupyterhub/kubespawner/blob/8766d364a29b9a688a268592fbee5e7a44190fad/kubespawner/reflector.py#L218-L223

It was added in #424. I'm not sure that it should be used in reflector._list_and_update at all, watcher implementation already has its own options: https://github.com/jupyterhub/kubespawner/blob/8766d364a29b9a688a268592fbee5e7a44190fad/kubespawner/reflector.py#L302-L304

ealyn commented 1 year ago

@dolfinus as the log I printed begining, the exception is raise at for p in initial_resources["items"]: https://github.com/jupyterhub/kubespawner/blob/a1a4a6f03c945d8633d565e5731afc44ee520caf/kubespawner/reflector.py#L228-L234

dolfinus commented 1 year ago

If await list_method(**kwargs) is called without _preload_content=False, and Kubernetes API returned 403 error, this function raises exception before for p in initial_resources["items"].