sgibson91
commented
2 years ago Problem
When requesting a server only results in the "Server Requested" log and does not proceed. Alex reported in the ticket this eventually changes to "Your server is stopping, you will be able to start is again once it has finished stopping."
Recreation
I logged in and tried to start a server myself and also got the "Server Requested" log and no further. I have not yet seen the "Your server is stopping..." message.
I activated kubectl access to the hub and looked at the events of my pod with kubectl describe pod jupyter-sgibson91 which revealed:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 17m gke.io/optimize-utilization-scheduler Successfully assigned ohw/jupyter-sgibson91 to gke-pilot-hubs-cluster-nb-ohw-e4ba9924-lj4s
Normal Pulled 17m kubelet Container image "ghcr.io/oceanhackweek/python:d98b914" already present on machine
Normal Created 17m kubelet Created container notebook
Normal Started 17m kubelet Started container notebookWhich indicates to me the server started successfully but none of the logs were streamed to the spawning page, and indeed the redirection to the server did not occur.
I tried stopping my server from the UI and then deleting the pod and this is when I see the "Your server is stopping..." message and appear to be stuck there.
All other pods in the namespace are running:
NAME READY STATUS RESTARTS AGE
api-ohw-dask-gateway-b45486c7b-62m9j 1/1 Running 0 12d
continuous-image-puller-44pjd 1/1 Running 0 47h
controller-ohw-dask-gateway-7f5dd9b6dd-ln9m7 1/1 Running 0 48d
hub-86b885f486-45pms 2/2 Running 1 5d1h
jupyter-abkfenris 1/1 Running 0 4h7m
jupyter-almacarolina 1/1 Running 0 46h
jupyter-anujjain2579 1/1 Running 0 10h
jupyter-clairedavies 1/1 Running 0 8h
jupyter-gmanuch 1/1 Running 0 17h
jupyter-jinjintwice 1/1 Running 0 9h
jupyter-leonardolaiolo 1/1 Running 0 8h
jupyter-noraloose 1/1 Running 0 19m
proxy-5bc58b774d-nmx9j 1/1 Running 0 34d
traefik-ohw-dask-gateway-694d9776f6-48hxg 1/1 Running 0 40d
abkfenris
yuvipanda
Summary
OceanHackWeek hub cannot start a server. Reported in https://2i2c.freshdesk.com/a/tickets/172
Impact on users
Hub unusable as no one can start a server.
Important information
Tasks and updates
After-action report template
``` # After-action report These sections should be filled out once we've resolved the incident and know what happened. They should focus on the knowledge we've gained and any improvements we should take. ## Timeline _A short list of dates / times and major updates, with links to relevant comments in the issue for more context._ All times in {{ most convenient timezone}}. - {{ yyyy-mm-dd }} - [Summary of first update](link to comment) - {{ yyyy-mm-dd }} - [Summary of another update](link to comment) - {{ yyyy-mm-dd }} - [Summary of final update](link to comment) ## What went wrong _Things that could have gone better. Ideally these should result in concrete action items that have GitHub issues created for them and linked to under Action items._ - Thing one - Thing two ## Where we got lucky _These are good things that happened to us but not because we had planned for them._ - Thing one - Thing two ## Follow-up actions _Every action item should have a GitHub issue (even a small skeleton of one) attached to it, so these do not get forgotten. These issues don't have to be in `infrastructure/`, they can be in other repositories._ ### Process improvements 1. {{ summary }} [link to github issue] 2. {{ summary }} [link to github issue] ### Documentation improvements 1. {{ summary }} [link to github issue] 2. {{ summary }} [link to github issue] ### Technical improvements 1. {{ summary }} [link to github issue] 2. {{ summary }} [link to github issue] ```