search

jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.02k stars 338 forks source link

Cannot bind traefik on a specific interface without breaking `tljh-config reload hub` #903

Closed nsurleraux-railnova closed 1 year ago

nsurleraux-railnova commented 1 year ago

Bug description

We want to avoid exposing our JupyterHub server on the internet. To do so, we have a file vpn_bind.toml in the traefik_config.d directory :

[entryPoints]
  [entryPoints.http]
  address = "VPN_IP:80"
  [entryPoints.https]
  address = "VPN_IP:443"

So the server is not listening anymore on the public interface but also on the loopback interface. This causes the check_hub_ready to break because 127.0.0.1 is hardcoded there.

Changing the address of the HTTP entrypoint to 127.0.0.1 only is not enough because requests tries to connect to 127.0.0.1:443 even if the port is 80 in the request (this is a mystery for me why it wants to try 443) :

>>> requests.get('http://127.0.0.1:80/hub/api', verify=False)
Traceback (most recent call last):
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connection.py", line 200, in _new_conn
    sock = connection.create_connection(
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/util/connection.py", line 85, in create_connection
    raise err
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connection.py", line 604, in connect
    self.sock = sock = self._new_conn()
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connection.py", line 215, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f90c6c72d90>: Failed to establish a new connection: [Errno 111] Connection refused

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/connectionpool.py", line 844, in urlopen
    retries = retries.increment(
  File "/opt/tljh/hub/lib/python3.8/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /hub/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f90c6c72d90>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/sessions.py", line 723, in send
    history = [resp for resp in gen]
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/sessions.py", line 723, in <listcomp>
    history = [resp for resp in gen]
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/sessions.py", line 266, in resolve_redirects
    resp = self.send(
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/opt/tljh/hub/lib/python3.8/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='127.0.0.1', port=443): Max retries exceeded with url: /hub/api (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f90c6c72d90>: Failed to establish a new connection: [Errno 111] Connection refused'))

The only way to make everything work is by not binding to a specific interface and listen to all interfaces, which is not desirable in our case.

Expected behaviour

Either the tljh-config reload hub command should detect the binding interface of Traefik or the config should allow to change the full check address instead of just the port.

Actual behaviour

The tljh-config reload hub command hangs because the check_ready_hub function is called in a loop.

How to reproduce

  1. Create a file in traefik_config.d to override the address of the http and https entrypoint so the traefik server is no longer listening on the loopback interface.
  2. tljh-config reload proxy works.
  3. netstat -tupln | grep 443 to check that the Traefik server is bound to your specific interface.
  4. tljh-config reload hub will hang.

Your personal set up

tljh==0.2.0 on ubuntu 20.04

Full environment ``` # paste output of `pip freeze` or `conda list` here ```
Configuration ```python # jupyterhub_config.py ```
Logs ``` # paste relevant logs here, if any ```
welcome[bot] commented 1 year ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

nsurleraux-railnova commented 1 year ago

The requests redirection may to be due to this : https://github.com/jupyterhub/the-littlest-jupyterhub/blob/main/tljh/traefik.toml.tpl#L30

minrk commented 1 year ago

adding a configurable ip to tljh-config makes sense.

The redirect to 443 should be correct because :80 only exists as a redirect to :443 when https is enabled.

nsurleraux-railnova commented 1 year ago

@minrk I wrote the code to do so. Could I assign you as a reviewer so you can check if it needs some polishing or anything else?