vmware-tanzu-labs / educates-training-platform

A platform for hosting interactive workshop environments in Kubernetes, or on top of a local container runtime.
https://docs.educates.dev
Apache License 2.0
63 stars 15 forks source link

Broken exception handling code in session-manager after logging cleanup. #490

Closed GrahamDumpleton closed 4 days ago

GrahamDumpleton commented 5 days ago

Describe the bug

INFO:educates.workshopsession:Workshop session creation request for lab-k8s-fundamentals-w03-s001 being retried, retries 3.
ERROR:kopf.objects:Handler 'workshop_session_create' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/http.py", line 437, in raise_for_status
    resp.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 409 Client Error: Conflict for url: https://10.96.0.1:443/api/v1/namespaces

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/src/handlers/workshopsession.py", line 737, in workshop_session_create
    pykube.Namespace(api, namespace_body).create()
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/objects.py", line 131, in create
    self.api.raise_for_status(r)
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/http.py", line 444, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: namespaces "lab-k8s-fundamentals-w03-s001" already exists

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/execution.py", line 276, in execute_handler_once
    result = await invoke_handler(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/execution.py", line 371, in invoke_handler
    result = await invocation.invoke(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/handlers/workshopsession.py", line 740, in workshop_session_create
    if e.code == 409:
       ^
UnboundLocalError: cannot access local variable 'e' where it is not associated with a value

Additional information

No response

GrahamDumpleton commented 5 days ago

Original error that triggered it from a bad workshop definition was:

ERROR:kopf.objects:Handler 'workshop_session_create' failed with an exception. Will retry.
Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/http.py", line 437, in raise_for_status
    resp.raise_for_status()
  File "/opt/app-root/venv/lib64/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 422 Client Error: Unprocessable Entity for url: https://10.96.0.1:443/apis/apps/v1/namespaces/lab-k8s-fundamentals-w03/deployments

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/execution.py", line 276, in execute_handler_once
    result = await invoke_handler(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/execution.py", line 371, in invoke_handler
    result = await invocation.invoke(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/venv/lib64/python3.12/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
    await asyncio.shield(future)  # slightly expensive: creates tasks
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/app-root/src/handlers/workshopsession.py", line 3109, in workshop_session_create
    pykube.Deployment(api, deployment_body).create()
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/objects.py", line 131, in create
    self.api.raise_for_status(r)
  File "/opt/app-root/venv/lib64/python3.12/site-packages/pykube/http.py", line 444, in raise_for_status
    raise HTTPError(resp.status_code, payload["message"])
pykube.exceptions.HTTPError: Deployment.apps "lab-k8s-fundamentals-w03-s001" is invalid: [spec.template.spec.containers[0].name: Invalid value: "<container-name>": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), spec.template.spec.containers[0].image: Required value]