search

jupyter / nb2kg

Other
73 stars 31 forks source link

ConnectionRefusedError: [Errno 61] Connection refused #20

Closed amangarg96 closed 5 years ago

amangarg96 commented 6 years ago

I'm trying to use nb2kg with JupyterHub to launch kernels on Spark using Jupyter Enterprise Gateway.

JupyterHub spawns Notebook Servers locally using the LocalSpawner class, while Jupyter Enterprise Gateway runs on a different server.

KG_URL was set as: export KG_URL=http://${JKG_HOST}:8888

nb2kg serverextension was activated by: jupyter serverextension enable --py nb2kg --sys-prefix

On starting JupyterHub and login in, I'm directed to the notebook server but cannot see any of the kernels.

Jupyter Hub shows the following log:

[I 2018-10-23 19:23:09.904 JupyterHub app:1906] JupyterHub is now running at http://:8000
[I 2018-10-23 19:25:05.039 JupyterHub log:158] 302 GET /user/aman.garg/api/sessions?_=1540296641591 -> /hub/user/aman.garg/api/sessions?_=1540296641591 (@::1) 1.19ms
[I 2018-10-23 19:25:05.042 JupyterHub log:158] 302 GET /user/aman.garg/api/terminals?_=1540296641592 -> /hub/user/aman.garg/api/terminals?_=1540296641592 (@::1) 0.87ms
[W 2018-10-23 19:25:05.056 JupyterHub log:158] 404 GET /hub/user/aman.garg/api/sessions?_=1540296641591 (aman.garg@::1) 10.12ms
[W 2018-10-23 19:25:05.062 JupyterHub log:158] 404 GET /hub/user/aman.garg/api/terminals?_=1540296641592 (aman.garg@::1) 14.83ms
[I 2018-10-23 19:25:06.348 JupyterHub log:158] 302 GET / -> /hub (@::1) 0.68ms
[I 2018-10-23 19:25:06.353 JupyterHub log:158] 302 GET /hub -> /hub/ (@::1) 0.46ms
[I 2018-10-23 19:25:06.363 JupyterHub log:158] 302 GET /hub/ -> /user/aman.garg/ (aman.garg@::1) 6.04ms
[I 2018-10-23 19:25:06.370 JupyterHub log:158] 302 GET /user/aman.garg/ -> /hub/user/aman.garg/ (@::1) 0.64ms
[D 2018-10-23 19:25:06.383 JupyterHub user:190] Creating <class 'jupyterhub.spawner.LocalProcessSpawner'> for aman.garg:
[D 2018-10-23 19:25:06.385 JupyterHub base:603] Initiating spawn for aman.garg
[D 2018-10-23 19:25:06.387 JupyterHub base:609] 0/100 concurrent spawns
[D 2018-10-23 19:25:06.387 JupyterHub base:612] 0 active servers
[I 2018-10-23 19:25:06.425 JupyterHub spawner:1100] Spawning jupyterhub-singleuser --port=57923
[D 2018-10-23 19:25:06.455 JupyterHub spawner:851] Polling subprocess every 30s
[I 2018-10-23 19:25:07.228 SingleUserNotebookApp extension:59] JupyterLab extension loaded from /Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterlab
[I 2018-10-23 19:25:07.228 SingleUserNotebookApp extension:60] JupyterLab application directory is /Users/aman.garg/anaconda2/envs/jupyterhub-nb/share/jupyter/lab
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:18] Loaded nb2kg extension
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernelspecs/(?P<kernel_name>[\\w\\.\\-%]+)$', <class 'nb2kg.handlers.KernelSpecHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernelspecs$', <class 'nb2kg.handlers.MainKernelSpecHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)/channels$', <class 'nb2kg.handlers.WebSocketChannelsHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)/(?P<action>restart|interrupt)$', <class 'nb2kg.handlers.KernelActionHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernels/(?P<kernel_id>\\w+-\\w+-\\w+-\\w+-\\w+)$', <class 'nb2kg.handlers.KernelHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp __init__:42] Overriding handler URLSpec('/user/aman.garg/api/kernels$', <class 'nb2kg.handlers.MainKernelHandler'>, kwargs=None, name=None)
[I 2018-10-23 19:25:07.233 SingleUserNotebookApp singleuser:405] Starting jupyterhub-singleuser server version 0.9.2
[I 2018-10-23 19:25:07.239 JupyterHub log:158] 200 GET /hub/api (@127.0.0.1) 1.29ms
[I 2018-10-23 19:25:07.240 SingleUserNotebookApp notebookapp:1685] Serving notebooks from local directory: /Users/aman.garg
[I 2018-10-23 19:25:07.240 SingleUserNotebookApp notebookapp:1685] The Jupyter Notebook is running at:
[I 2018-10-23 19:25:07.240 SingleUserNotebookApp notebookapp:1685] http://127.0.0.1:57923/user/aman.garg/
[I 2018-10-23 19:25:07.240 SingleUserNotebookApp notebookapp:1686] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2018-10-23 19:25:07.516 SingleUserNotebookApp log:158] 302 GET /user/aman.garg/ -> /user/aman.garg/tree? (@127.0.0.1) 0.73ms
[D 2018-10-23 19:25:07.517 JupyterHub utils:188] Server at http://127.0.0.1:57923/user/aman.garg/ responded with 302
[D 2018-10-23 19:25:07.517 JupyterHub _version:53] jupyterhub and jupyterhub-singleuser both on version 0.9.2
[I 2018-10-23 19:25:07.517 JupyterHub base:628] User aman.garg took 1.131 seconds to start
[I 2018-10-23 19:25:07.517 JupyterHub proxy:242] Adding user aman.garg to proxy /user/aman.garg/ => http://127.0.0.1:57923
[D 2018-10-23 19:25:07.517 JupyterHub proxy:678] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/aman.garg
19:25:07.519 - info: [ConfigProxy] Adding route /user/aman.garg -> http://127.0.0.1:57923
19:25:07.519 - info: [ConfigProxy] 201 POST /api/routes/user/aman.garg
[I 2018-10-23 19:25:07.520 JupyterHub log:158] 302 GET /hub/user/aman.garg/ -> /user/aman.garg/?redirects=1 (aman.garg@::1) 1143.19ms
[I 2018-10-23 19:25:07.530 SingleUserNotebookApp log:158] 302 GET /user/aman.garg/?redirects=1 -> /user/aman.garg/tree?redirects=1 (@::1) 0.72ms
[I 2018-10-23 19:25:07.538 SingleUserNotebookApp log:158] 302 GET /user/aman.garg/tree?redirects=1 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-aman.garg&redirect_uri=%2Fuser%2Faman.garg%2Foauth_callback&response_type=code&state=[secret] (@::1) 2.25ms
[I 2018-10-23 19:25:07.559 JupyterHub log:158] 302 GET /hub/api/oauth2/authorize?client_id=jupyterhub-user-aman.garg&redirect_uri=%2Fuser%2Faman.garg%2Foauth_callback&response_type=code&state=[secret] -> /user/aman.garg/oauth_callback?code=[secret]&state=[secret] (aman.garg@::1) 13.95ms
[I 2018-10-23 19:25:07.597 JupyterHub log:158] 200 POST /hub/api/oauth2/token (aman.garg@127.0.0.1) 16.73ms
[I 2018-10-23 19:25:07.612 SingleUserNotebookApp auth:875] Logged-in user {'kind': 'user', 'name': 'aman.garg', 'admin': True, 'groups': [], 'server': '/user/aman.garg/', 'pending': None, 'created': '2018-08-22T10:10:37.285516Z', 'last_activity': '2018-10-23T13:55:07.606547Z', 'servers': None}
[I 2018-10-23 19:25:07.613 SingleUserNotebookApp log:158] 302 GET /user/aman.garg/oauth_callback?code=[secret]&state=[secret] -> /user/aman.garg/tree?redirects=1 (@::1) 47.19ms
[I 2018-10-23 19:25:07.617 JupyterHub log:158] 200 GET /hub/api/authorizations/token/[secret] (aman.garg@127.0.0.1) 13.10ms
[W 2018-10-23 19:25:07.628 SingleUserNotebookApp auth:303] Failed to check authorization: [400] Bad Request
[W 2018-10-23 19:25:07.628 SingleUserNotebookApp auth:304] {"error": "invalid_request", "error_description": "Invalid authorization code parameter"}
[W 2018-10-23 19:25:07.628 SingleUserNotebookApp web:1667] 500 GET /user/aman.garg/oauth_callback?code=0a7d1416-eeb3-4c43-98c7-1c759cf841ae&state=eyJ1dWlkIjogIjEwMjhhYjk3N2M5ODRlZWJhYWE2YzVmODU4ODc4ZTQ5IiwgIm5leHRfdXJsIjogIi91c2VyL2FtYW4uZ2FyZy90cmVlP3JlZGlyZWN0cz0xIn0 (::1): Failed to check authorization
[W 2018-10-23 19:25:07.631 JupyterHub log:158] 400 POST /hub/api/oauth2/token (aman.garg@127.0.0.1) 7.86ms
[E 2018-10-23 19:25:07.661 SingleUserNotebookApp log:150] {
      "X-Forwarded-Host": "localhost:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::1",
      "Cookie": "jupyterhub-user-aman.garg=2|1:0|10:1540296641|25:jupyterhub-user-aman.garg|48:MmIyNDlkZjgtZmNjOS00YmNlLWExNDAtZDM0YWU2NmZmMzEy|10c183c186ac14918143c94d57f5f7d59fe229e8b56cf753b470b6eefdf933e5; jupyterhub-user-aman.garg-oauth-state=2|1:0|10:1540302907|37:jupyterhub-user-aman.garg-oauth-state|164:ZXlKMWRXbGtJam9nSWpFd01qaGhZamszTjJNNU9EUmxaV0poWVdFMll6Vm1PRFU0T0RjNFpUUTVJaXdnSW01bGVIUmZkWEpzSWpvZ0lpOTFjMlZ5TDJGdFlXNHVaMkZ5Wnk5MGNtVmxQM0psWkdseVpXTjBjejB4SW4w|5283e06eec026eb0c00c5a935efb80c34256c4fa11314d29b578ff4b59d2db3b; username-localhost-8890=\"2|1:0|10:1539596743|23:username-localhost-8890|44:YzRhNTQyMDM5NGE2NDczZWEzNGMzY2NhZDUwNzVmOGY=|cd1a0ca00774f149cef6d2a0b6ee82535764f29a7028ea7a8215fbc3bec56e35\"; username-localhost-8889=\"2|1:0|10:1539718031|23:username-localhost-8889|44:OGE4NjViODZkYzMxNDA4Mjk4MGM1ZjQ4YTFkMGY5MmM=|65ee55b39625d31a9bbf1f6b6d9f279fcf06672dc2938a061b4477c2851f4867\"; _xsrf=2|f40dc673|d91f73aa05e138328d9fe2d79b5fe7e4|1540213551; jupyterhub-session-id=42658b79a2164516a080a171757d41e8; username-localhost-8888=\"2|1:0|10:1540291437|23:username-localhost-8888|44:YmI2OTI5NWZiYmNjNGYwMWI2ODUwN2MxMGUxN2M2MDU=|8a2547a844b069d5741cb235b72c0e4689efbef85bbe46825ba0ebc4c2e15cc2\"",
      "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
      "Accept-Encoding": "gzip, deflate, br",
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
      "Upgrade-Insecure-Requests": "1",
      "Purpose": "prefetch",
      "Connection": "close",
      "Host": "localhost:8000"
    }
[E 2018-10-23 19:25:07.662 SingleUserNotebookApp log:158] 500 GET /user/aman.garg/oauth_callback?code=[secret]&state=[secret] (@::1) 45.08ms
[I 2018-10-23 19:25:07.708 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/tree?redirects=1 (aman.garg@::1) 46.00ms
[I 2018-10-23 19:25:07.887 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/api/config/tree?_=1540302907831 (aman.garg@::1) 2.79ms
[I 2018-10-23 19:25:07.890 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/api/config/common?_=1540302907832 (aman.garg@::1) 2.72ms
[I 2018-10-23 19:25:07.892 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/api/terminals?_=1540302907833 (aman.garg@::1) 0.88ms
[I 2018-10-23 19:25:07.895 SingleUserNotebookApp managers:283] Request list kernel specs at: /api/kernelspecs
[I 2018-10-23 19:25:07.898 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/api/sessions?_=1540302907834 (aman.garg@::1) 1.87ms
[I 2018-10-23 19:25:07.900 SingleUserNotebookApp log:158] 200 GET /user/aman.garg/api/terminals?_=1540302907835 (aman.garg@::1) 0.86ms
[E 2018-10-23 19:25:07.902 SingleUserNotebookApp web:1670] Uncaught exception GET /user/aman.garg/api/kernelspecs (::1)
    HTTPServerRequest(protocol='http', host='localhost:8000', method='GET', uri='/user/aman.garg/api/kernelspecs', version='HTTP/1.1', remote_ip='::1')
    Traceback (most recent call last):
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/nb2kg/handlers.py", line 324, in get
        kernel_specs = yield gen.maybe_future(ksm.list_kernel_specs())
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/nb2kg/managers.py", line 284, in list_kernel_specs
        response = yield fetch_kg(self.kernelspecs_endpoint, method='GET')
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/nb2kg/managers.py", line 64, in fetch_kg
        response = yield client.fetch(url, **kwargs)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
    ConnectionRefusedError: [Errno 61] Connection refused
[W 2018-10-23 19:25:07.904 SingleUserNotebookApp handlers:564] Unhandled error
[E 2018-10-23 19:25:07.905 SingleUserNotebookApp log:150] {
      "X-Forwarded-Host": "localhost:8000",
      "X-Forwarded-Proto": "http",
      "X-Forwarded-Port": "8000",
      "X-Forwarded-For": "::1",
      "Cookie": "jupyterhub-user-aman.garg=2|1:0|10:1540302907|25:jupyterhub-user-aman.garg|48:OWIyOTFmZGItNmEzYy00MGVlLWI0OTgtNzA1MzY0ZTRhZmY1|7e9a43be118d1b873db359a0b31f0e57c62bf1383648f9450e20788d17c5bfb2; username-localhost-8890=\"2|1:0|10:1539596743|23:username-localhost-8890|44:YzRhNTQyMDM5NGE2NDczZWEzNGMzY2NhZDUwNzVmOGY=|cd1a0ca00774f149cef6d2a0b6ee82535764f29a7028ea7a8215fbc3bec56e35\"; username-localhost-8889=\"2|1:0|10:1539718031|23:username-localhost-8889|44:OGE4NjViODZkYzMxNDA4Mjk4MGM1ZjQ4YTFkMGY5MmM=|65ee55b39625d31a9bbf1f6b6d9f279fcf06672dc2938a061b4477c2851f4867\"; _xsrf=2|f40dc673|d91f73aa05e138328d9fe2d79b5fe7e4|1540213551; jupyterhub-session-id=42658b79a2164516a080a171757d41e8; username-localhost-8888=\"2|1:0|10:1540291437|23:username-localhost-8888|44:YmI2OTI5NWZiYmNjNGYwMWI2ODUwN2MxMGUxN2M2MDU=|8a2547a844b069d5741cb235b72c0e4689efbef85bbe46825ba0ebc4c2e15cc2\"",
      "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "http://localhost:8000/user/aman.garg/tree",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36",
      "X-Xsrftoken": "2|f40dc673|d91f73aa05e138328d9fe2d79b5fe7e4|1540213551",
      "X-Requested-With": "XMLHttpRequest",
      "Accept": "*/*",
      "Connection": "close",
      "Host": "localhost:8000"
    }
[E 2018-10-23 19:25:07.905 SingleUserNotebookApp log:158] 500 GET /user/aman.garg/api/kernelspecs (aman.garg@::1) 9.93ms

What could be the issue for connection getting refused? My setup is working for individual notebook servers, this issue only arises when launching notebook server using JupyterHub

kevin-bates commented 6 years ago

Thanks for the issue. I typically see ECONNREFUSED when attempting to hit a gateway server and EG is not running. When called from my mac where EG is located (but not running) on a different machine, I get the following:

[I 15:33:40.191 NotebookApp] Request list kernel specs at: /api/kernelspecs
[E 15:33:42.387 NotebookApp] Uncaught exception GET /api/kernelspecs (172.17.0.1)
    HTTPServerRequest(protocol='http', host='localhost:9003', method='GET', uri='/api/kernelspecs', version='HTTP/1.1', remote_ip='172.17.0.1')
    Traceback (most recent call last):
      File "/opt/conda/lib/python3.6/site-packages/tornado/web.py", line 1592, in _execute
        result = yield result
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/opt/conda/lib/python3.6/site-packages/nb2kg/handlers.py", line 324, in get
        kernel_specs = yield gen.maybe_future(ksm.list_kernel_specs())
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/opt/conda/lib/python3.6/site-packages/nb2kg/managers.py", line 284, in list_kernel_specs
        response = yield fetch_kg(self.kernelspecs_endpoint, method='GET')
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
        yielded = self.gen.throw(*exc_info)
      File "/opt/conda/lib/python3.6/site-packages/nb2kg/managers.py", line 64, in fetch_kg
        response = yield client.fetch(url, **kwargs)
      File "/opt/conda/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
        value = future.result()
    ConnectionRefusedError: [Errno 111] Connection refused

The errno values are different but the error is the same. I suspect that's more about mac vs. linux.

I'm assuming you've confirmed your EG server is running on ${JKG_HOST} at port 8888 - but just in case. Also, when you ran the similar scenario without Hub, were all other aspects (namely topology) the same? You might also take a look at this blog post. Although its specific to kubernetes, the Hub configuration details will be similar.

amangarg96 commented 6 years ago

Hey,

Yes, the EG server was running on ${JKG_HOST} at port 8888, and I was able to run the similar scenario without JupyterHub. (See #435 for the configuration of that setup)

One thing which is different in this setup though, is that in my current setup the JupyterHub (And the Notebook Servers spawned by JupyterHub) are running on a Mac, while the Enterprise Gateway is running on a Linux machine.

Could this be causing issues?

kevin-bates commented 6 years ago

No, that shouldn't matter. My typical config is Notebook/NB2KG on mac and EG on linux.

So when you say the Hub config is different, have you tried just running NB on mac? Sounds like there is a topology difference between your NB/EG and Hub/EG envs and I would suggest eliminating that difference first. Might there be any kind of firewall between your systems in this env vs. the one w/o Hub?

amangarg96 commented 6 years ago

Yes, I should try the NB-EG config with Mac first.

Let me try that and get back with the observations.

kevin-bates commented 6 years ago

@amangarg96 - did you make any progress on this?

Fyi, I ran into another instance of ConnectionRefusedError but this one was relative to errno 61 (rather than 111). Since I'm running in different conda envs, I wonder if the errno differences are due to different underlying packages.

amangarg96 commented 6 years ago

Hey, as discussed I tested the NB-EG config first and it was running successfully.

But when I tried running JupyterHub, I got the following error - 

[C 2018-10-26 12:13:53.576 JupyterHub app:1861] Failed to start proxy
    Traceback (most recent call last):
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/app.py", line 1859, in start
        await self.proxy.start()
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/proxy.py", line 596, in start
        await server.wait_up(1)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/utils.py", line 165, in wait_for_server
        timeout=timeout
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at <hostname>:8000 didn't respond in 1 seconds

I'm not able to reproduce the ConnectionRefusedError now.

Here's the full log - 

sudo jupyterhub -f /Users/aman.garg/.jupyter/jupyterhub_config.py --debug

Password:
[D 2018-10-26 12:13:40.864 JupyterHub application:555] Looking for /Users/aman.garg/.jupyter/jupyterhub_config in /Users/aman.garg
[D 2018-10-26 12:13:40.874 JupyterHub application:577] Loaded config file: /Users/aman.garg/.jupyter/jupyterhub_config.py
[I 2018-10-26 12:13:40.876 JupyterHub app:1667] Using Authenticator: jupyterhub.auth.PAMAuthenticator-0.9.2
[I 2018-10-26 12:13:40.876 JupyterHub app:1667] Using Spawner: jupyterhub.spawner.LocalProcessSpawner-0.9.2
[I 2018-10-26 12:13:40.983 JupyterHub app:1014] Loading cookie_secret from /Users/aman.garg/jupyterhub_cookie_secret
[D 2018-10-26 12:13:40.988 JupyterHub app:1069] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2018-10-26 12:13:41.294 JupyterHub orm:685] database schema version found: 896818069c98
[I 2018-10-26 12:13:41.299 JupyterHub proxy:431] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2018-10-26 12:13:41.300 JupyterHub app:1171] No admin users, admin interface will be unavailable.
[W 2018-10-26 12:13:41.300 JupyterHub app:1172] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2018-10-26 12:13:41.300 JupyterHub app:1199] Not using whitelist. Any authenticated user will be allowed.
[D 2018-10-26 12:13:41.333 JupyterHub app:1471] Loading state for aman.garg from db
[D 2018-10-26 12:13:41.335 JupyterHub app:1487] Loaded users:
    aman.garg admin
[I 2018-10-26 12:13:41.344 JupyterHub app:1849] Hub API listening on http://127.0.0.1:8081/hub/
[W 2018-10-26 12:13:41.344 JupyterHub proxy:472] Found proxy pid file: /Users/aman.garg/jupyterhub-proxy.pid
[W 2018-10-26 12:13:41.346 JupyterHub proxy:489] Proxy still running at pid=401
[W 2018-10-26 12:13:42.350 JupyterHub proxy:504] Stopped proxy at pid=401
[D 2018-10-26 12:13:42.350 JupyterHub proxy:520] Removing proxy pid file jupyterhub-proxy.pid
[W 2018-10-26 12:13:42.352 JupyterHub proxy:560] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2018-10-26 12:13:42.352 JupyterHub proxy:562] Starting proxy @ http://:8000
[D 2018-10-26 12:13:42.352 JupyterHub proxy:563] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://127.0.0.1:8081/hub/error']
[D 2018-10-26 12:13:42.357 JupyterHub proxy:512] Writing proxy pid file: jupyterhub-proxy.pid
12:13:43.103 - info: [ConfigProxy] Proxying http://*:8000 to (no default)
12:13:43.106 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[C 2018-10-26 12:13:53.576 JupyterHub app:1861] Failed to start proxy
    Traceback (most recent call last):
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/app.py", line 1859, in start
        await self.proxy.start()
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/proxy.py", line 596, in start
        await server.wait_up(1)
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/utils.py", line 165, in wait_for_server
        timeout=timeout
      File "/Users/aman.garg/anaconda2/envs/jupyterhub-nb/lib/python3.6/site-packages/jupyterhub/utils.py", line 155, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at blr-af-dbl48h2.fkipl.flipkart.com:8000 didn't respond in 1 seconds

[D 2018-10-26 12:13:53.593 JupyterHub application:647] Exiting application: jupyterhub
kevin-bates commented 6 years ago

This strikes me as a Hub issue only and you're unable to start an NB server (as I'm sure you're aware). Unfortunately, I'm not familiar with troubleshooting Hub, so you might need to run this particular scenario passed folks in Hub. Their gitter channel is fairly active, so that's probably your best bet.

amangarg96 commented 6 years ago

Thanks, I'll get in touch with JupyterHub guys for this issue.

amangarg96 commented 5 years ago

I have started working on the JupyterHub and Enterprise Gateway again, and I am running into the same ConnectionRefusedError: [Errno 61] Connection refused error with the same configuration.

However, when I ran the Jupyter Enterprise Gateway on the local (the same machine where JupyterHub is running and Notebook Servers are being launched) and pointed KG_URL to that, it's working.

And again, the same configuration without JupyterHub is working fine too.

amangarg96 commented 5 years ago

Hey, I just realized that when I was running JEG on my local, it was using the default KG_URL (127.0.0.1:8888). When I ran the JEG on a different port (locally), I still got ConnectionRefusedError: [Errno 61] Connection refused

As @kevin-bates mentioned, the error was because the NB server was attempting to hit a gateway server and EG wasn't running.

Maybe the environment variable KG_URL is not getting passed to NB server.

I'll try passing the KG_URL=<FQDN of Gateway Endpoint> through JupyterHub instead.

amangarg96 commented 5 years ago

Passing the environment variable through JupyterHub's config file (jupyterhub_config.py) solved the issue.

To pass the environment variable, I added KG_URL in c.Spawner.env_keep and then launched Jupyterhub.