dashboard creation yields 404

[ricky-lim]

Describe the bug

During starting up a dashboard, it yields 404

To Reproduce

Local jupyterhub setup with minikube: https://github.com/ricky-lim/jupyterhub-minikube/tree/fastapi main.py: https://github.com/ricky-lim/jupyterhub-minikube/blob/fastapi/examples/main.py

Screenshots https://drive.google.com/file/d/13u8ZBx7nADi-nJI3Z9k6dIF8vaWaQLFh/view?usp=sharing

Any tweaks could you advise to avoid 404 during the start-up ?

Thanks in advance. I look forward to hearing from you

[danlester]

I have seen this before - I think it almost always happens with a Kubernetes system actually, and it happens (without ContainDS Dashboards) when you make a 'named server' (even a regular Jupyter server).

I think it must be a problem with KubeSpawner itself, perhaps incorrectly determining that the server is ready, although I'm not sure exactly what conditions cause the problem. I would like to take a look at some point - I don't think it is on the KubeSpawner project's radar because not too many people use named servers outside of ContainDS I feel.

Do you eventually find that reloading accesses the dashboard correctly?

[ricky-lim]

Hi Dan,

Yes, it eventually reloads :). Unfortunately for most of users, this causes confusion.

In addition to that, as I also implemented culling-activity service from jupyterhub. Therefore for users to access the culled-dashboard would give even worse dashboard experience.

For regular jupyter-server, such as jupyter-ril, it does not yield 404. So far it starts up as expected.

It does not seem to cause non-functional bugs however this glitch would provide negative user accessibility, imho.

[danlester]

I completely agree that it is confusing for users.

How about a regular named Jupyter server (not My Server)? If you type 'test' in the Named Server box on the home page, do you initially get the 404 problem with jupyter-ril-test?

[danlester]

Actually I have been trying to reproduce the issue using the latest helm chart for z2jh (not your repo yet, sorry!), and haven't seen the problem yet today... I wonder if something has been fixed in KubeSpawner.

Would you mind trying to update to the latest helm chart 0.11.1 to see if that helps you too? (Instead of 0.9.0)

[ricky-lim]

Hi Dan,

Just tested on my minikube setup, indeed with 0.11.1, the issue is being resolved

[danlester]

That's great to hear, thanks for trying it out!

[ricky-lim]

Unfortunately, I have still experienced that if I recreated the same dashboard during the initialization, it still yields 404.

However if I killed first the pod with kubectl delete pod, it does initialize with success.

Still puzzled with this phenomena

[danlester]

Ultimately, there will be times where the pod is unavailable, but the proxy that intercepts /user/dan/dash-test traffic hasn't yet been notified about the latest state, or vice versa. I haven't been able to reproduce it yet, but I might have to artificially slow things down to understand what's happening. It may be that KubeSpawner needs to do more to keep these in sync, or it's possible that Dashboards isn't quite leaving KubeSpawner in the right state.

How about with named regular Jupyter servers? Are you able to reproduce the problem there, e.g. if you kill pods unexpectedly? If we can pin this down to 'all servers' rather than 'only dashboard servers' it will be incredibly helpful in working further on it.

[danlester]

Ricky, I have done some work that attempts to connect to the KubeSpawner in more detail, hoping to wait until the dashboard is definitely ready before allowing the redirect...

If you would be kind enough to test, that would be really helpful:

The version on PyPI, or tag in GitHub is 0.5.0b1

Thanks,

Dan

[ricky-lim]

Hi Dan,

Sorry for my late reply. I just tested on the minikube and also on-premise rancher.

It does now works without 404. Thanks for the update.

[danlester]

Fantastic, thanks for your time testing this! Let me know anything else that comes up, but this should find its way to the next full release.

[ricky-lim]

Any plan yet for the 0.5.0 release?

[danlester]

Should be released over the next couple of weeks, but ultimately no reason to be afraid of 0.5.0b2 if that version works for you! (Or do you need it inside some automated Docker image or similar?)

[danlester]

@ricky-lim I had a bit of time to release 0.5.0 today - hope that helps!

[ricky-lim]

Thank you danlester for your great work on the latest release. 🙏