search

Momentlabs / athenaeum

Jupyter Notebooks as a service
1 stars 0 forks source link

proxy is failing when jlab gets started up from the browser by changing the URL #38

Open jdrivas opened 6 years ago

jdrivas commented 6 years ago

Stack trace from proxy. 0:49:44.522 - debug: [ConfigProxy] PROXY WEB / to http://10.11.242.174:8081 20:49:44.586 - debug: [ConfigProxy] PROXY WEB /hub to http://10.11.242.174:8081 20:49:44.632 - debug: [ConfigProxy] PROXY WEB /hub/ to http://10.11.242.174:8081 20:49:44.680 - debug: [ConfigProxy] PROXY WEB /hub/login to http://10.11.242.174:8081 20:49:44.795 - debug: [ConfigProxy] PROXY WEB /hub/static/css/style.min.css?v=dd1df30ccc6c4d3e9705d78012d25b57 to http://10.11.242.174:8081 20:49:44.796 - debug: [ConfigProxy] PROXY WEB /hub/static/components/requirejs/require.js?v=e7199843dfd445bb66ec816e98a03214 to http://10.11.242.174:8081 20:49:44.797 - debug: [ConfigProxy] PROXY WEB /hub/static/components/jquery/dist/jquery.min.js?v=a09e13ee94d51c524b7e2a728c7d4039 to http://10.11.242.174:8081 20:49:44.798 - debug: [ConfigProxy] PROXY WEB /hub/static/components/bootstrap/dist/js/bootstrap.min.js?v=5869c96cc8f19086aee625d670d741f9 to http://10.11.242.174:8081 20:49:44.800 - debug: [ConfigProxy] PROXY WEB /hub/logo to http://10.11.242.174:8081 20:49:47.206 - debug: [ConfigProxy] PROXY WEB /favicon.ico to http://10.11.242.174:8081 20:49:48.412 - debug: [ConfigProxy] PROXY WEB /hub/oauth_login?next= to http://10.11.242.174:8081 20:50:02.254 - debug: [ConfigProxy] PROXY WEB /hub/oauth_callback?code=Y4x7iM1TBwswzLTI&state=eyJzdGF0ZV9pZCI6ICJjOGQ0ZDJhZDU3NmE0MjMzYTFjZTcwOGUwMmJmYmQ5OCIsICJuZXh0X3VybCI6ICIifQ%3D%3D to http://10.11.242.174:8081 20:50:02.578 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/ to http://10.8.0.13:8888 20:50:02.626 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/tree? to http://10.8.0.13:8888 20:50:02.694 - debug: [ConfigProxy] PROXY WEB /hub/api/oauth2/authorize?client_id=jupyterhub-user-auth0_58b8af3b20e74746f356e4c9&redirect_uri=%2Fuser%2Fauth0_58b8af3b20e74746f356e4c9%2Foauth_callback&response_type=code&state=eyJ1dWlkIjogIjE5MDU5OGU4M2I2ZTRjMmQ4ZGRjODM2ODQ1ODc2NGIyIiwgIm5leHRfdXJsIjogIi91c2VyL2F1dGgwXzU4YjhhZjNiMjBlNzQ3NDZmMzU2ZTRjOS90cmVlPyJ9 to http://10.11.242.174:8081 20:50:02.791 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/oauth_callback?code=e5c0599b-5ceb-48d2-9d67-7d30332f2d7c&state=eyJ1dWlkIjogIjE5MDU5OGU4M2I2ZTRjMmQ4ZGRjODM2ODQ1ODc2NGIyIiwgIm5leHRfdXJsIjogIi91c2VyL2F1dGgwXzU4YjhhZjNiMjBlNzQ3NDZmMzU2ZTRjOS90cmVlPyJ9 to http://10.8.0.13:8888 20:50:02.933 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/tree? to http://10.8.0.13:8888 20:50:03.022 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=3c2a865c832a1322285c55c6ed99abb2 to http://10.8.0.13:8888 20:50:03.023 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=7afb461de36accb1aa133a1710f5bc56 to http://10.8.0.13:8888 20:50:03.037 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/style/style.min.css?v=4b4b8cb1e49605137f77fed041f8922b to http://10.8.0.13:8888 20:50:03.038 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/custom/custom.css to http://10.8.0.13:8888 20:50:03.054 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe to http://10.8.0.13:8888 20:50:03.055 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/preact/index.js?v=00a2fac73c670ce39ac53d26640eb542 to http://10.8.0.13:8888 20:50:03.056 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/proptypes/index.js?v=c40890eb04df9811fcc4d47e53a29604 to http://10.8.0.13:8888 20:50:03.056 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/preact-compat/index.js?v=aea8f6660e54b18ace8d84a9b9654c1c to http://10.8.0.13:8888 20:50:03.057 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/requirejs/require.js?v=951f856e81496aaeec2e71a1c2c0d51f to http://10.8.0.13:8888 20:50:03.058 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/tree/js/main.min.js?v=c8a3b910b6156957e4347dea8ff2f6a3 to http://10.8.0.13:8888 20:50:03.938 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/services/contents.js?v=20181106181417 to http://10.8.0.13:8888 20:50:03.954 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/base/images/favicon.ico?v=97c6417ed01bdc0ae3ef32ae4894fd03 to http://10.8.0.13:8888 20:50:04.006 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/config/tree?=1542055840622 to http://10.8.0.13:8888 20:50:04.007 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/config/common?=1542055840623 to http://10.8.0.13:8888 20:50:04.019 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/terminals?=1542055840624 to http://10.8.0.13:8888 20:50:04.020 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/api/kernelspecs to http://10.8.0.13:8888 20:50:04.021 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/sessions?=1542055840625 to http://10.8.0.13:8888 20:50:04.021 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/terminals?=1542055840626 to http://10.8.0.13:8888 20:50:04.041 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/font-awesome/fonts/fontawesome-webfont.woff2?v=4.7.0 to http://10.8.0.13:8888 20:50:04.167 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/custom/custom.js?v=20181106181417 to http://10.8.0.13:8888 20:50:04.288 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/contents?type=directory&=1542055840627 to http://10.8.0.13:8888 20:50:12.467 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/contents/Follow%20Along%20Books?type=directory&=1542055840628 to http://10.8.0.13:8888 20:50:16.790 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/api/contents/Follow%20Along%20Books to http://10.8.0.13:8888 20:50:17.138 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/notebooks/Follow%20Along%20Books/Untitled.ipynb?kernel_name=python3 to http://10.8.0.13:8888 20:50:18.671 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured to http://10.8.0.13:8888 20:50:18.672 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/bootstrap-tour/build/css/bootstrap-tour.min.css?v=d0b3c2fce6056a2ddd5a4513762a94c4 to http://10.8.0.13:8888 20:50:18.672 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/codemirror/lib/codemirror.css?v=288352df06a67ee35003b0981da414ac to http://10.8.0.13:8888 20:50:18.673 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/notebook/css/override.css?v=e6f18013b8771987812e992b38ec3318 to http://10.8.0.13:8888 20:50:18.673 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/text-encoding/lib/encoding.js?v=d5bb0fc9ffeff7d98a69bb83daa51052 to http://10.8.0.13:8888 20:50:18.674 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/custom/custom.css to http://10.8.0.13:8888 20:50:18.675 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/notebook/js/main.min.js?v=fc2a61bcf6f091823db7fb553a838e7f to http://10.8.0.13:8888 20:50:19.002 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/config/TeX-AMS-MML_HTMLorMML-full.js?V=2.7.5 to http://10.8.0.13:8888 20:50:19.703 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/config/Safe.js?V=2.7.5 to http://10.8.0.13:8888 20:50:19.812 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/base/images/favicon-notebook.ico?v=9e5fb7f8501d94094806320e718be6b3 to http://10.8.0.13:8888 20:50:19.819 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/config/notebook?=1542055856443 to http://10.8.0.13:8888 20:50:19.823 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/config/common?=1542055856444 to http://10.8.0.13:8888 20:50:19.844 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/extensions/MathML/content-mathml.js?V=2.7.5 to http://10.8.0.13:8888 20:50:19.860 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/api/kernelspecs to http://10.8.0.13:8888 20:50:19.910 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/contents/Follow%20Along%20Books/Untitled.ipynb?type=notebook&=1542055856445 to http://10.8.0.13:8888 20:50:19.911 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/custom/custom.js?v=20181106181417 to http://10.8.0.13:8888 20:50:19.914 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?V=2.7.5 to http://10.8.0.13:8888 20:50:19.923 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/components/MathJax/extensions/Safe.js?V=2.7.5 to http://10.8.0.13:8888 20:50:19.926 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/nbextensions/jupyter-js-widgets/extension.js?v=20181106181417 to http://10.8.0.13:8888 20:50:20.104 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/api/sessions to http://10.8.0.13:8888 20:50:20.107 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/contents/Follow%20Along%20Books/Untitled.ipynb/checkpoints?=1542055856446 to http://10.8.0.13:8888 20:50:20.107 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/kernelspecs/python3/logo-64x64.png to http://10.8.0.13:8888 20:50:20.454 - debug: [ConfigProxy] PROXY WS /user/auth0_58b8af3b20e74746f356e4c9/api/kernels/340600b8-3b18-49fd-bf85-5fe8e39a6f73/channels?session_id=8fea2379e8714a138ebcd7296b5ca8ef to http://10.8.0.13:8888 20:50:21.332 - debug: [ConfigProxy] PROXY WEB /user/auth0_58b8af3b20e74746f356e4c9/static/base/images/favicon-busy-1.ico to http://10.8.0.13:8888 20:50:26.382 - error: [ConfigProxy] 503 GET /user/auth0_58b8af3b20e74746f356e4c9/api/kernels/340600b8-3b18-49fd-bf85-5fe8e39a6f73/channels?session_id=8fea2379e8714a138ebcd7296b5ca8ef socket hang up 20:50:26.383 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.onOutgoingError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/ws-incoming.js:149:9) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at Socket.socketOnEnd (_http_client.js:346:9) at emitNone (events.js:91:20) at Socket.emit (events.js:185:7) 20:50:26.384 - error: [ConfigProxy] 503 GET /user/auth0_58b8af3b20e74746f356e4c9/api/kernels/340600b8-3b18-49fd-bf85-5fe8e39a6f73/channels?session_id=8fea2379e8714a138ebcd7296b5ca8ef socket hang up 20:50:26.385 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.onOutgoingError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/ws-incoming.js:149:9) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at Socket.socketCloseListener (_http_client.js:286:9) at emitOne (events.js:101:20) at Socket.emit (events.js:188:7) 20:50:27.937 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/sessions?=1542055840629 to http://10.8.0.13:8888 20:50:27.941 - debug: [ConfigProxy] PROXY WEB /user/auth058b8af3b20e74746f356e4c9/api/terminals?=1542055840630 to http://10.8.0.13:8888 20:50:30.947 - error: [ConfigProxy] 503 GET /user/auth058b8af3b20e74746f356e4c9/api/sessions?=1542055840629 Error: connect EHOSTUNREACH 10.8.0.13:8888 at Object.exports._errnoException (util.js:1020:11) at exports._exceptionWithHostPort (util.js:1043:20) at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14) 20:50:30.950 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.proxyError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:154:11) at emitOne (events.js:101:20) at ClientRequest.emit (events.js:188:7) at Socket.socketErrorListener (_http_client.js:310:9) at emitOne (events.js:96:13) at Socket.emit (events.js:188:7) 20:50:30.951 - error: [ConfigProxy] 503 GET /user/auth058b8af3b20e74746f356e4c9/api/terminals?=1542055840630 Error: connect EHOSTUNREACH 10.8.0.13:8888 at Object.exports._errnoException (util.js:1020:11) at exports._exceptionWithHostPort (util.js:1043:20) at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14) 20:50:30.951 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.proxyError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:154:11) at emitOne (events.js:101:20) at ClientRequest.emit (events.js:188:7) at Socket.socketErrorListener (_http_client.js:310:9) at emitOne (events.js:96:13) at Socket.emit (events.js:188:7) 20:50:30.951 - error: [ConfigProxy] 503 GET /user/auth058b8af3b20e74746f356e4c9/api/terminals?=1542055840630 socket hang up 20:50:30.951 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.proxyError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:154:11) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at Socket.socketCloseListener (_http_client.js:286:9) at emitOne (events.js:101:20) at Socket.emit (events.js:188:7) 20:50:30.952 - error: [ConfigProxy] 503 GET /user/auth058b8af3b20e74746f356e4c9/api/sessions?=1542055840629 socket hang up 20:50:30.952 - error: [ConfigProxy] Uncaught Exception Error: Protocol "10.11.242.174:" not supported. Expected "http:" at new ClientRequest (_http_client.js:54:11) at Object.exports.request (http.js:31:10) at ConfigurableProxy.handleProxyError (/usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:430:31) at /usr/local/lib/node_modules/configurable-http-proxy/lib/configproxy.js:500:14 at ClientRequest.proxyError (/usr/local/lib/node_modules/configurable-http-proxy/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:154:11) at emitOne (events.js:96:13) at ClientRequest.emit (events.js:188:7) at Socket.socketCloseListener (_http_client.js:286:9) at emitOne (events.js:101:20) at Socket.emit (events.js:188:7)

jdrivas commented 6 years ago

The fix requires to:

  1. Delete the running notebook server
  2. restart the hub, which then deletes the proxy paths the notebook server.

I'm assuming that something goes wrong with jl which possibly restarts the notebook server in a way that doesn't inform the proxy. Though ....?

jdrivas commented 6 years ago

This is not just jlab. We get disconnected in a non-jlab environment as well.

jdrivas commented 6 years ago

Now it seems that, if the notebook crashes (and why should it crash) then restarts, the proxy gets quite confused.

jdrivas commented 6 years ago

Killing the server via JHub main page works.