search

jupyterhub / configurable-http-proxy

node-http-proxy plus a REST API
BSD 3-Clause "New" or "Revised" License
242 stars 130 forks source link

EKS container pods unable to communicate with proxy #501

Closed bbry89 closed 1 year ago

bbry89 commented 1 year ago

Bug description

I have z2jh set up on AWS EKS using configurable-http-proxy as the proxy. Recently, I have seen jupyterhub being slow to load and then failing with ERR_EMPTY_RESPONSE message after a few minutes of attempting to load. If jupyterhub does get loaded, it will pop up a message with "Server unavailable or unreachable" after just a few minutes. At this point, Jupyter will not respond to any operations or commands given.

I have seen failures in the browser console with the same ERR_EMPTY_RESPONSE message in response to api calls: https://host:port/user/username1/api/sessions?content=1&1695851658265. However, kubectl logs ... and kubectl describe pods ... does not show any indications of errors from kubernetes side.

The only fix that I have found is to restart our proxy server, which is a kubernetes pod running configurable-http-proxy. Immediately after restarting, all issues seen with jupyterhub clients disappears.

How to reproduce

The issues randomly occurs. I have not been able to track down what causes this issue to reproduce it.

Expected behaviour

Jupyter client should be able to connect to kubernetes pods without issues

Actual behaviour

All Jupyter client API calls fail with ERR_EMPTY_RESPONSE, rendering it completely unresponsive to any actions taken by a user until the proxy is restarted.

Your personal set up

jupyterhub==3.1.0 jupyterlab==3.5.2 notebook==6.2.0 configurable-http-proxy==4.5.4 kubernetes==1.24

Full environment ``` # paste output of `pip freeze` or `conda list` here ```
Configuration ```python # jupyterhub_config.py ```
Logs ``` # paste relevant logs here, if any ```
welcome[bot] commented 1 year ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

manics commented 1 year ago

This is likely to be a problem with your Kubernetes cluster, rather than a bug with JupyterHub/Z2JH/Configurable http proxy.

For example, I've seen problem related to the EKS networking plugin before. Unfortunately there's no easy solution, you may need to start debugging EKS at quite a low level. If you need more help your best option is to use the Jupyter community forum and see if anyone else has an idea https://discourse.jupyter.org/