search

nesi / nesi-ood-jupyterlab-app

OOD JupyterLab app for Analysis Environment (Beta)
0 stars 0 forks source link

Ondemand loses connection with ongoing Jupyter session #5

Open ndjones opened 2 months ago

ndjones commented 2 months ago

After some period of time, ondemand lost its handle on a running jupyter session, reporting:

Jupyter Lab - staging (jupyterlab-t0x4kmix) Undetermined

Your session has entered a bad state. Feel free to contact support for further information.

chrisdjscott commented 2 months ago

When this happens does the app still show up as running under interactive sessions: https://163-7-144-39.sslip.io/pun/sys/dashboard/batch_connect/sessions ?

ndjones commented 2 months ago

the error above is copied from the session box for the related Jupyter session under interactive sessions. I didn't copy all the text, just the two indications of an issue.

lbrick commented 2 months ago

I believe I have recreated the issue.

It looks related to the OIDC token expiration, if I log in and out then click the "restart web server" once or twice the token gets reissued and the card returns to green.

I dont believe this is related to Jupyter as i have a feeling you can also recreate this with any app running

chrisdjscott commented 2 months ago

did we have this issue on the training environment too and had to adjust some timeout values in keycloak?

chrisdjscott commented 2 months ago

maybe this bit https://github.com/nesi/training-environment/blob/main/roles/keycloak/tasks/main.yml#L151-L154

lbrick commented 2 months ago

Unsure if we did have the issue on the training environment

The Keycloak we are now using is controlled/maintained by CNI so will need to discuss with them if we can increase the SSO timeouts for tokens

lbrick commented 2 months ago

Yuriy and I have managed to resolve this.

We are investigating a few other options, however this is fixed in the current test environment and is related to OOD => kubernetes configuration more then the OOD application itself

I will leave this issue open for now till Yuriy and I have tried a few other ways.