welcome[bot]
commented
2 years ago Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
minrk
connorourke
Bug description
I have a jupyterhub server up and running, with users able to log on, as am I. However some users get a 503 error and are unable to log on, and it is not temporary - the problem is persistent.
If I remove the user account, and get them to log on again and recreate the account the problem goes away.
Expected behaviour
The users should be able to log on
Actual behaviour
The user gets a 503 error
How to reproduce
Your personal set up
Installed using conda with users authenticating using Azure AAD
Version(s):
jupyterhub 1.4.2 Python 3.8.12
Full environment
Configuration
```python import os import sys from oauthenticator.azuread import LocalAzureAdOAuthenticator from oauthenticator import LocalAuthenticator import systemdspawner # Debug level c.Application.log_level = 'DEBUG' # Network & security c.JupyterHub.port = 443 c.JupyterHub.ssl_key = '/etc/letsencrypt/live/addresshere/privkey.pem' c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/addresshere/fullchain.pem' c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80'] # Spawning c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner' c.SystemdSpawner.user_workingdir = '/jupyter/{USERNAME}' c.SystemdSpawner.username_template = '{USERNAME}' c.SystemdSpawner.cmd=["jupyter-labhub"] # Cull idle users (new section) c.JupyterHub.services = [ { 'name': 'idle-culler', 'admin': True, 'command': [ sys.executable, '-m', 'jupyterhub_idle_culler', '--timeout=3600' ], } ] c.SystemdSpawner.start_timeout = 400 c.SystemdSpawner.http_timeout = 400 c.SystemdSpawner.poll_interval = 400 c.JupyterHub.tornado_settings = { 'slow_spawn_timeout': 10 } c.SystemdSpawner.mem_limit = '0.5G' c.SystemdSpawner.cpu_limit = 0.45 # Authenication c.JupyterHub.authenticator_class = LocalAzureAdOAuthenticator c.LocalAzureAdOAuthenticator.tenant_id = os.environ.get('AAD_TENANT_ID') c.LocalAzureAdOAuthenticator.oauth_callback_url = os.environ.get('AAD_CALLBACK') c.LocalAzureAdOAuthenticator.client_id = os.environ.get('AAD_APP_CLIENT_ID') c.LocalAzureAdOAuthenticator.client_secret = os.environ.get('AAD_APP_CLIENT_SECRET') c.LocalAzureAdOAuthenticator.username_claim = 'unique_name' ```Logs
I get lots of 503 errors in the logs like this: ``` Apr 4 14:09:21 res-dev-jh-maths-vm bash[3774]: [I 2022-04-04 14:09:21.273 SingleUserLabApp kernelmanager:187] Kernel started: e8724899-ce29-4d1d-a553-ade835c5c6c0 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.273 JupyterHub pages:652] No template for 503 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.274 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%2F022770c5-6b15-49a5-a080-72ec66cf9beb%2Fchannels%3Fsession_id%3De7d76c93-fd50-4e68-acd9-d5b979174393 (@127.0.0.1) 2.01ms Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.344 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/kernels/d124afa2-663c-4d6d-ad1b-6d7f3fa779c3/channels?session_id=fc2a6e85-d8aa-4625-8e59-67324186f2cb connect ECONNREFUSED 127.0.0.1:56899 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.345 JupyterHub pages:652] No template for 503 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.347 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%2Fd124afa2-663c-4d6d-ad1b-6d7f3fa779c3%2Fchannels%3Fsession_id%3Dfc2a6e85-d8aa-4625-8e59-67324186f2cb (@127.0.0.1) 2.05ms Apr 4 14:09:21 res-dev-jh-maths-vm bash[3774]: [I 2022-04-04 14:09:21.363 SingleUserLabApp log:189] 201 POST /user/jupyter-epo24/api/sessions?1649081359814 (jupyter-epo24@::ffff:138.38.220.241) 999.93ms Apr 4 14:09:21 res-dev-jh-maths-vm bash[3432]: [I 2022-04-04 14:09:21.370 SingleUserLabApp log:189] 204 PUT /user/jupyter-eod34/lab/api/workspaces/default?1649081361881 (jupyter-eod34@::ffff:138.38.244.181) 3.54ms Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.545 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/sessions?1649081357304 connect ECONNREFUSED 127.0.0.1:56899 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.547 JupyterHub pages:652] No template for 503 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.548 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fsessions%3F1649081357304 (@127.0.0.1) 2.02ms Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.569 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/kernels?1649081357305 connect ECONNREFUSED 127.0.0.1:56899 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.571 JupyterHub pages:652] No template for 503 Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.572 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%3F1649081357305 (@127.0.0.1) 1.74ms ```