Unauthorized error while scheduling new workers

[ops-gaurav]

Describe the issue:

We have been using Dask for a while now but suddenly started getting Unauthorized error while scheduling new workers through Jupyter notebook.

Minimal Complete Verifiable Example:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
    result = await invoke_handler(
  File "/usr/local/lib/python3.8/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
    result = await invocation.invoke(
  File "/usr/local/lib/python3.8/site-packages/kopf/_core/actions/invocation.py", line 116, in invoke
    result = await fn(**kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/operator/controller/controller.py", line 311, in daskworkergroup_create
    await daskworkergroup_update(
  File "/usr/local/lib/python3.8/site-packages/dask_kubernetes/operator/controller/controller.py", line 401, in daskworkergroup_update
    cluster = await customobjectsapi.get_namespaced_custom_object(
  File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
    raise e
  File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
    response_data = await self.request(
  File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 193, in GET
    return (await self.request("GET", url,
  File "/usr/local/lib/python3.8/site-packages/kubernetes_asyncio/client/rest.py", line 187, in request
    raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (401)
Reason: Unauthorized
HTTP response headers: <CIMultiDictProxy('Audit-Id': 'a9d1d013-c336-43a1-a026-e0a3193fcb19', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Fri, 16 Aug 2024 12:23:51 GMT', 'Content-Length': '129')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

Anything else we need to know?:

Environment:

• Dask version: 2023.1.1
• Python version: >= 3.8
• Operating System: Debian GNU/Linux 11 (bullseye) (Docker)
• Install method (conda, pip, source): pip

[jacobtomlinson]

It looks like you're using quite an old version there. We haven't supported Python 3.8 for a while and the kubernetes_asyncio client dependency that is throwingt he exception has been completely replaced with kr8s.

I would suggest updating to a new version and seeing if the issues still happen.

[ops-gaurav]

@jacobtomlinson I am not sure what the issue was but we restarted the dask-operator and the error went away. I am closing this ticket and I will plan on migrating to latest version and re-open if I face similar issue.