jupyterhub / kubespawner

Kubernetes spawner for JupyterHub
https://jupyterhub-kubespawner.readthedocs.io
BSD 3-Clause "New" or "Revised" License
536 stars 301 forks source link

improve efficiency of reflector #755

Closed juliantaylor closed 12 months ago

juliantaylor commented 1 year ago

The reflector performs a _list_and_update every watch timeout (default 10s) requesting most recent data from the apiserver via the default resource_version="".

This requires the apiserver to retrieve the latest data from the backing store (etcd) which can be very expensive in large clusters. For example if omit_namespace is set the full cluster pod listings require several seconds retrieve all data from the etcd in clusters with more then 10000 pods. The label selector filtering is only done in the apiserver.

As the _list_and_update is used for watches the listing should set the resource version and resource version match to tell the apiserver that its locally cached data is sufficient. The initial listing will fetch any data the apiserver currently has to prepopulate the watch, the watch itself will then fetch whatever is remaining from the etcd.

The subsequent relistings after the watch timeouts use their last received resource version in the same way.

For details see https://kubernetes.io/docs/reference/using-api/api-concepts/#resource-versions

Closes gh-753

welcome[bot] commented 1 year ago

Thanks for submitting your first pull request! You are awesome! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please make sure you followed the pull request template, as this will help us review your contribution more quickly. welcome You can meet the other Jovyans by joining our Discourse forum. There is also a intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

welcome[bot] commented 12 months ago

Congrats on your first merged pull request in this project! :tada: congrats Thank you for contributing, we are very proud of you! :heart:

yuvipanda commented 12 months ago

Thanks, @juliantaylor!