search

jupyter-on-openshift / jupyterhub-quickstart

OpenShift compatible version of the JupyterHub application.
Apache License 2.0
101 stars 107 forks source link

Upstream failure verifying auth token: [502] Host not found #27

Closed Mohamed-0-Hassan closed 4 years ago

Mohamed-0-Hassan commented 4 years ago

502 : Bad Gateway Upstream failure verifying auth token: [502] Host not found

I'm trying to run Jupyterhub on openshift with the following command.

oc new-app --template jupyterhub-quickstart --param APPLICATION_NAME=app_name --param GIT_REPOSITORY_URL=https://github.com/jakevdp/PythonDataScienceHandbook --param BUILDER_IMAGE=s2i-minimal-notebook:3.5

Below the output from the log file. [W 2019-09-27 22:00:24.537 SingleUserNotebookApp configurable:168] Config option open_browsernot recognized bySingleUserNotebookApp. Did you mean browser? [I 2019-09-27 22:00:25.611 SingleUserNotebookApp extension:168] JupyterLab extension loaded from /opt/app-root/lib/python3.5/site-packages/jupyterlab [I 2019-09-27 22:00:25.612 SingleUserNotebookApp extension:169] JupyterLab application directory is /opt/app-root/share/jupyter/lab [I 2019-09-27 22:00:25.614 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.6 [I 2019-09-27 22:00:26.638 SingleUserNotebookApp log:158] 302 GET /user/2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc/ -> /user/2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc/tree? (@102.101.111.17) 0.92ms [I 2019-09-27 22:00:26.693 SingleUserNotebookApp log:158] 302 GET /user/2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc/?redirects=1 -> /user/2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc/tree?redirects=1 (@10.201.12.1) 0.82ms [W 2019-09-27 22:00:26.738 SingleUserNotebookApp auth:586] Detected unused OAuth state cookies [I 2019-09-27 22:00:26.740 SingleUserNotebookApp log:158] 302 GET /user/2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc/tree?redirects=1 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-2fa17b8d-d67e-4e4f-bf0c-62123e db0bdc&response_type=code&redirect_uri=%2Fuser%2F2fa17b8d-d67e-4e4f-bf0c-62123edb0bdc%2Foauth_callback&state=[secret] (@10.201.16.1) 2.76ms [E 2019-09-27 22:00:26.886 SingleUserNotebookApp auth:299] Upstream failure verifying auth token: [502] Host not found [E 2019-09-27 22:00:26.886 SingleUserNotebookApp auth:300]

GrahamDumpleton commented 4 years ago

What version of OpenShift are you trying to use?

This is sometimes bad cookies cached in the browser if JupyterHub has been restarted a number of times. Trying deleting cookies in the browser for that specific host.

Mohamed-0-Hassan commented 4 years ago

I'm running "openshift v3.11.59"
I have delete the cookies cached in my browser. This didn't help.

GrahamDumpleton commented 4 years ago

Are you using Minishift, OpenShift Online, or your own cluster?

Mohamed-0-Hassan commented 4 years ago

i'm running my own cluster:

oc v3.11.0+0cbc58b kubernetes v1.11.0+d4cacc0 features: Basic-Auth SSPI Kerberos SPNEGO Server https://master......l:443 openshift v3.11.59 kubernetes v1.11.0+d4cacc0

Below the complete pod log. `oc logs pod/astream01-nb-67a8a45d-2d9ae8-2d40c3-2dbd87-2d2f77ab765f32

[W 2019-09-29 06:28:33.069 SingleUserNotebookApp web:1667] 502 GET /user/67a8a45d-9ae8-40c3-bd87-2f77ab765f32/oauth_callback?code=f3fb049c-c9cc-464a-92bb-6de8b294a7db&state=eyJ1dWlkIjogImMyZmZhMzk4NTEwNDQxMjQ5ZGRiZWQyZWY2Y2Q4MjRiIiwgIm5leHRfdXJsIjogIi91c2VyLzY3YThhNDVkLTlhZTgtNDBjMy1iZDg3LTJmNzdhYjc2NWYzMi90cmVlP3JlZGlyZWN0cz0xIn0 (10.201.16.1): Failed to check authorization (upstream problem) [E 2019-09-29 06:28:33.158 SingleUserNotebookApp log:158] 502 GET /user/67a8a45d-9ae8-40c3-bd87-2f77ab765f32/oauth_callback?code=[secret]&state=[secret] (@10.201.16.1) 102.33ms [I 2019-09-29 06:28:37.712 SingleUserNotebookApp notebookapp:1774] Serving notebooks from local directory: /opt/app-root/src [I 2019-09-29 06:28:37.713 SingleUserNotebookApp notebookapp:1774] The Jupyter Notebook is running at: [I 2019-09-29 06:28:37.713 SingleUserNotebookApp notebookapp:1774] http://(astream01-nb-67a8a45d-2d9ae8-2d40c3-2dbd87-2d2f77ab765f32 or 127.0.0.1):8080/user/67a8a45d-9ae8-40c3-bd87-2f77ab765f32/ [I 2019-09-29 06:28:37.713 SingleUserNotebookApp notebookapp:1775] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).`

GrahamDumpleton commented 4 years ago

What happens is that the notebook instance needs to make a web request back to the JupyterHub instance. This would be done using an internal HTTP request within the cluster, by using the service name of the JupyterHub instance as the host name.

Your cluster seems to be configured in a way that it is injecting HTTP proxy environment variables (or something like that), into all pods, such that even what should be an internal web request from a pod to a service in the same project, is routed via the proxy. That is then failing.

Is the configuration of cluster somehow inject something to force the use of a proxy for all pods? Are there any proxy related environment variables set on the notebook pod?

What happens if you access the pod for the notebook using a terminal and run:

curl -v http://astream01:8081/

If curl is not available, try using wget instead.

Mohamed-0-Hassan commented 4 years ago

Hi Gramham,

You are right it's some how injecting http proxy env.

below the output of the wget command inside the pod.

`oc exec astream01-nb-d98676e0-2dde5f-2d4c2c-2db442-2d1a12583bde91 -- wget -v http://astream01:8081/ --2019-09-29 09:32:08-- http://astream01:8081/ Resolving proxy01.example.com (proxy01.example.com)... 11.21.21.28 Connecting to proxy01.example.com (proxy01.example.com)|11.21.21.28|:8080... connected. Proxy request sent, awaiting response... 502 Host not found 2019-09-29 09:32:13 ERROR 502: Host not found.

command terminated with exit code 8 `

GrahamDumpleton commented 4 years ago

I don't know if this will work or not, but edit the config map for the JupyterHub instead and in the entry for jupyterhub_config.py add:

c.Spawner.environment['http_proxy'] = ''
c.Spawner.environment['https_proxy'] = ''

Do this for any proxy environment variables injected into the pod.

Once changed, trigger a new deployment of JupyterHub and try again.

GrahamDumpleton commented 4 years ago

What you possibly want to use is:

c.Spawner.environment['NO_PROXY'] = 'astream01'
Mohamed-0-Hassan commented 4 years ago

this leads to a erro on the pod

[E 2019-10-03 10:56:46.244 JupyterHub app:1958] c.Spawner.environment['NO_PROXY'] = 'astream01' TypeError: 'LazyConfigValue' object does not support item assignment

GrahamDumpleton commented 4 years ago

Change it to:

c.Spawner.environment = dict()
c.Spawner.environment['NO_PROXY'] = 'astream01'
Mohamed-0-Hassan commented 4 years ago

Hi Graham,

The TypError is solved but, still running into the same error as before.

[I 2019-10-03 11:57:39.635 SingleUserNotebookApp log:158] 302 GET /user/b0c790d6-dd20-48e9-b7da-85693ebbd2e4/tree?redirects=1 -> /hub/api/oauth2/authorize?response_type=code&client_id=jupyterhub-user-b0c790d6-dd20-48e9-b7da-85693ebbd2e4&redirect_uri=%2Fuser%2Fb0c790d6-dd20-48e9-b7da-85693ebbd2e4%2Foauth_callback&state=[secret] (@10.201.12.1) 2.47ms [E 2019-10-03 11:57:39.758 SingleUserNotebookApp auth:299] Upstream failure verifying auth token: [502] Host not found

GrahamDumpleton commented 4 years ago

Instead of 'astream01' try '*'.

GrahamDumpleton commented 4 years ago

It isn't the same error though is it. So it means the NO_PROXY possibly worked doesn't it. Maybe this time it is the browser cookies.

GrahamDumpleton commented 4 years ago

Ignore that, not seeing 'Host not found' on next line. Try '*'. Verify that the environment variable is set on the pod.

GrahamDumpleton commented 4 years ago

Also try setting 'no_proxy' in lower case as well.

Mohamed-0-Hassan commented 4 years ago

Hi Graham,

Thanks a lot. The solution is to set the no_proxy. c.Spawner.environment['no_proxy'] = '<APPLICATION_NAME or *>'. The uppercase one didn't work.

GrahamDumpleton commented 4 years ago

Great, at least know a workaround for it. Will have to think about whether should add something more permanent related to this. Maybe see if anyone else encounters it first.

Mohamed-0-Hassan commented 4 years ago

This works fine with the minimal notebook but, when is try to launch a tensorflow, scipy or a custom notebook. I run into the same error.

502 : Bad Gateway Failed to check authorization (upstream problem)

GrahamDumpleton commented 4 years ago

Are you trying to use the Jupyter project images, rather than the OpenShift based ones?

Even so, interpretation of no_proxy should be handled by the Python libraries, which should be the same for both. So it shouldn't make a difference.

Have you checked the pod definition during the period it exists to verify it is being set for the container?

Mohamed-0-Hassan commented 4 years ago

I'm using the openshift ones from the following links. oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/build-configs/s2i-scipy-notebook.json oc create -f https://raw.githubusercontent.com/jupyter-on-openshift/jupyter-notebooks/master/build-configs/s2i-tensorflow-notebook.json

c.Spawner.environment['no_proxy'] = 'astream' c.KubeSpawner.profile_list = [ { 'display_name': 'Minimal Notebook (Classic Python 3.6)', 'default': True, 'kubespawner_override': { 'image_spec': 's2i-minimal-notebook:3.6' } }, { 'display_name': 'Minimal Notebook (Tensorflow JupyterLab Python 3.6)', 'kubespawner_override': { 'image_spec': 's2i-tensorflow-notebook:3.6', 'environment': { 'JUPYTER_ENABLE_LAB': 'true' } } }, { 'display_name': 'Minimal Notebook (SCIPY JupyterLab Python 3.6)', 'kubespawner_override': { 'image_spec': 's2i-scipy-notebook:3.6', 'environment': { 'JUPYTER_ENABLE_LAB': 'true' } } },

GrahamDumpleton commented 4 years ago

There really should be no difference. Did you verify that the environment variables were being set on the pod? Did you try again with *?

Mohamed-0-Hassan commented 4 years ago

I tried the wildcard in no_proxy and also removed all the notebook images and download it again. The redirect still worked for the minimal notebook only.

oc new-app --template jupyterhub-deployer   --param APPLICATION_NAME=data   --param JUPYTERHUB_CONFIG="`cat jp.py`"  --param NOTEBOOK_IMAGE=s2i-scipy-notebook:3.6

oc new-app --template jupyterhub-deployer   --param APPLICATION_NAME=data   --param JUPYTERHUB_CONFIG="`cat jp.py`"  --param NOTEBOOK_IMAGE=s2i-minimal-notebook:3.6

some output from get -v http://astream:8081/

scipy

--2019-10-25 11:26:31--  http://astream:8081/
Resolving proxy.example.local (proxy.example.local)... 10.24.221.28
Connecting to proxy.example.local (proxy.example.local)|10.24.211.28|:8080... connected.
Proxy request sent, awaiting response... 502 Host not found
2019-10-25 11:26:31 ERROR 502: Host not found.

minimal notebook

--2019-10-25 11:23:51--  http://astream:8081/
Resolving astream (astream)... 10.24.221.28
Connecting to astream(astream)|10.201.150.196|:8081... connected.
HTTP request sent, awaiting response... 302 Found
Location: /hub [following]
--2019-10-25 11:23:51--  http://astream:8081/hub
Reusing existing connection to astream:8081.
HTTP request sent, awaiting response... 302 Found
Location: /hub/ [following]
--2019-10-25 11:23:51--  http://astream:8081/hub/
Reusing existing connection to astream:8081.
HTTP request sent, awaiting response... 302 Found
Location: /hub/login [following]
--2019-10-25 11:23:51--  http://astream:8081/hub/login
Reusing existing connection to astream:8081.
HTTP request sent, awaiting response... 200 OK
Length: 4687 (4.6K) [text/html]
Saving to: ‘index.html.1’

100%[===================================================================================================================================================>] 4,687       --.-K/s   in 0s

2019-10-25 11:23:51 (583 MB/s) - ‘index.html.1’ saved [4687/4687]
Mohamed-0-Hassan commented 4 years ago

Fixed by creating a new build and set the no_proxy an NO_PROXY in the file /.s2i/environment.