search

dask / dask-kubernetes

Native Kubernetes integration for Dask
https://kubernetes.dask.org
BSD 3-Clause "New" or "Revised" License
311 stars 148 forks source link

Can not connect to k8s websocket deployed in Rancher #851

Open dbalabka opened 10 months ago

dbalabka commented 10 months ago

Describe the issue: Deploy k8s cluster in a rancher. Try to connect with default settings. You will see:

2024-01-02 21:19:05,453 - asyncio - ERROR - Task exception was never retrieved
future: <Task finished name='Task-26' coro=<PortForward._sync_sockets() done, defined at /home/torinaki/src/demand-forecasting/.venv/lib/python3.10/site-packages/kr8s/_portforward.py:168> exception=AssertionError('Only absolute URLs without path part are supported')>
Traceback (most recent call last):
  File ".../.venv/lib/python3.10/site-packages/kr8s/_portforward.py", line 171, in _sync_sockets
    async with self._connect_websocket() as ws:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File ".../.venv/lib/python3.10/site-packages/kr8s/_portforward.py", line 150, in _connect_websocket
    async with self.pod.api.open_websocket(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File ".../.venv/lib/python3.10/site-packages/kr8s/_api.py", line 184, in open_websocket
    async with aiohttp.ClientSession(
  File ".../.venv/lib/python3.10/site-packages/aiohttp/client.py", line 261, in __init__
    self._base_url.origin() == self._base_url
AssertionError: Only absolute URLs without path part are supported

because Rancher provides the following URL to connect to websockets: https://rancher.example.com/k8s/clusters/c-m-4wx4gzpx

Minimal Complete Verifiable Example:

TBD

Anything else we need to know?:

Environment:

dbalabka commented 10 months ago

To fix the issue, aiohttp needs to support the path in base_url: https://github.com/aio-libs/aiohttp/issues/6647

It works for me in my patched version.

jacobtomlinson commented 10 months ago

Would you mind opening an issue upstream in kr8s for this? Preferably with a small example we can use to reproduce the issue (I know we would need Rancher but the snippet that reproduces on your end would still be useful).

dbalabka commented 10 months ago

@jacobtomlinson , I think the simplest way to reproduce the problem is to put k8s API behind the proxy and add an extra path (proxy http://127.0.0.1/somepath to http://127.0.0.1) and adjust kubectl settings. The Nginx reverse proxy_pass rule can help:

location /somepath {
  proxy_pass http://127.0.0.1
}
jacobtomlinson commented 10 months ago

It looks like kubectl can handle this for you.

$ kubectl proxy --api-prefix=/somepath/
Starting to serve on 127.0.0.1:8001
$ curl localhost:8001/somepath/version
{
  "major": "1",
  "minor": "29",
  "gitVersion": "v1.29.0",
  "gitCommit": "3f7a50f38688eb332e2a1b013678c6435d539ae6",
  "gitTreeState": "clean",
  "buildDate": "2023-12-14T19:18:17Z",
  "goVersion": "go1.21.5",
  "compiler": "gc",
  "platform": "linux/amd64"
}
jacobtomlinson commented 10 months ago

While digging into this I've noticed some other strange behaviour when the Kubernetes API is proxied with an extra path. I opened https://github.com/kr8s-org/kr8s/issues/284 to track.