Connection refused - Githubissues

b-trav commented 6 years ago

My issue may be similar to @debsahu on #1.

I followed instructions as described here.

I am attempting to get colaboratory working from a browser on my laptop using a different computer as the local runtime - let's call this computer "local_server", which happens to have an ip address of 10.1.1.3 on my local LAN. The browser that I am using is the chrome (Version 67.0.3396.99 (Official Build) (64-bit))

Colaboratory works no problems with hosted runtime.
Colaboratory works no problems with local runtime as my laptop (on my laptop I have jupyter running on port 8888).
I have jupyter set up on my local_server, also on port 8888
Colaboratory works no problem with local runtime when I am on using chrome on the local_server machine

I now want to work on colaboratory on my laptop using local_server as the local runtime

To get ssh forwarding is working, I run this command on my laptop:
```
ssh -L 8889:10.1.1.3:8888 10.1.1.3
```

In the jupyter_notebook_config.py file on the local_server machine I set

c.NotebookApp.ip = '10.1.1.3'

Now I can go to http://localhost:8889/tree on my laptop and jupyter works but colaboratory does not. I get the following error from jupyter:

[E 10:22:43.257 NotebookApp] Uncaught error when proxying request
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/jupyter_http_over_ws/handlers.py", line 151, in on_message
    response.rethrow()
  File "/usr/local/lib/python3.4/dist-packages/tornado/httpclient.py", line 652, in rethrow
    raise self.error
ConnectionRefusedError: [Errno 111] Connection refused

What am I doing wrong? Any suggestions would be appreciated. Thanks.

blois commented 6 years ago

Is this following the instructions at https://research.google.com/colaboratory/local-runtimes.html?

Which browser? Any additional error message from the browser dev tools console?

b-trav commented 6 years ago

My apologies, I started submitting an issue, but got distracted. I did not realise the (empty) issue was posted.

debsahu commented 6 years ago

Did you remove/disable ad blocker?

b-trav commented 6 years ago

I have ad blocker running on chrome, but I have disabled it for colab.research.google.com.

b-trav commented 6 years ago

Changing to

c.NotebookApp.ip = '*'

does not help.

ClaudioCimarelli commented 6 years ago

I have the same issue. I am using a remote server by SSH tunneling the localhost port ( similar situation to using Google Cloud as described at the bottom of your guide).
Opening Jupyter on the local browser has no problem and the connection got accepted from the server, but Colab get refused. Note that I'm using --no-browser option.

blois commented 6 years ago

What SSH client are you using? Since you mention that the connection works locally, just not when SSH'd to another machine, it sounds like the issue may be at that layer.

Can you view and execute Jupyter notebooks over the SSH connection, following the URL which is displayed when launching Jupyter?

ClaudioCimarelli commented 6 years ago

The problem is the firewall actually. I tested by opening a WS between my notebook (simulating a Colab request) and the remote server and I got the same problem. Anyone incurring in this issue should probably check their Firewall rules. My intention was to use my laptop as an intermediary for the connection between Colab and the server with a firewall. Because it is blocking all request of connection through w.s., I am wondering if I could forward the request of Colab through ssh instead...

jiho commented 6 years ago

tl;dr

On SERVER, use

jupyter notebook --no-browser \
  --NotebookApp.allow_origin='https://colab.research.google.com' \
  --port=8888 \
  --NotebookApp.token='' --NotebookApp.password='' \
  --NotebookApp.disable_check_xsrf=True

On LAPTOP use

ssh ME@SERVER -L 8888:localhost:8888 -N

Connect to colab from LAPTOP and use Local runtime on port 8888 and it works.

Long version

I used to have the exact same error:

On SERVER

jupyter notebook --no-browser \
  --NotebookApp.allow_origin='https://colab.research.google.com' \
  --port=8888 \
  --NotebookApp.token='' --NotebookApp.password=''

(I disabled all security to be sure it was not a problem).

On LAPTOP

ssh -F/dev/null ME@SERVER -L 9999:localhost:8888  -N

(I disabled the usual ssh configuration with -F/dev/null just to be sure, you may not need it)

On LAPTOP I can connect to http://localhost:9999/ and I get the jupyter interface. From colab, trying to connect to the Local runtime on port 9999 gives "Unable to connect" and the following message on SERVER:

[E 18:13:45.474 NotebookApp] Uncaught error when proxying request
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/jupyter_http_over_ws/handlers.py", line 151, in on_message
        response.rethrow()
      File "/usr/local/lib/python3.5/dist-packages/tornado/httpclient.py", line 652, in rethrow
        raise self.error
    ConnectionRefusedError: [Errno 111] Connection refused

However, setting the port-forward to the same port (8888 rather than 9999) gave me the following error on SERVER:

[W 18:13:24.141 NotebookApp] 403 POST /api/sessions (::1): '_xsrf' argument missing from POST
[W 18:13:24.141 NotebookApp] '_xsrf' argument missing from POST
[W 18:13:24.142 NotebookApp] 403 POST /api/sessions (::1) 2.40ms referer=None

Which led me to add the --NotebookApp.disable_check_xsrf=True option when running the jupyter notebook.

So it seems related to a port error. I have now idea why.

Also, I would welcome tips on how to allow connection without removing all security (no token, no password, etc.).

tim-tx commented 6 years ago

I think the instructions from @jiho should be on the official instructions for local runtimes: https://research.google.com/colaboratory/local-runtimes.html

This is just a direct connection to a local server, no ssh tunnel. I had to use empty values for token and password as above for out-of-the-box jupyter and no local config files in $HOME/.jupyter. Without these parameters, navigating to https://127.0.0.1:8888/tree redirects to https://127.0.0.1:8888/login?next=%2Ftree%3F where I'm told token authentication is enabled and asked to enter a token or create a password. There's no redirect or token notification if you navigate to https://localhost:8888. This is jupyter version 4.4.0, I guess token authentication is enabled by default. I didn't see anything that looked like a system-wide config included in my distribution's jupyter-notebook package. Also had to use disable_check_xsrf=True as above.

I second the need to have colaboratory be able to connect to servers that use authentication. Not sure if this is the right place to request a feature.

Hoiy commented 6 years ago

Thanks @jiho, your solution saved my day.

I have similar issues as @b-trav trying to connect colab script to a gcp vm.

It works well with colab to local server and local to gcp remote server following the instruction in this link.

However, without @jiho's fix, I got the following errors connecting colab to gcp vm server:

[W 10:57:39.707 NotebookApp] Forbidden
[W 10:57:39.707 NotebookApp] 403 GET /api/kernelspecs (::1) 1.22ms referer=None

It would be great to have colab connecting to gcp vm directly without local port forwarding (having a really slow connection). Well, I am happy for now :)

kwlzn commented 5 years ago

for cases of connecting to ssh-tunneled notebook server instances, it seems like there's an implicit requirement here that the local tunnel port is the exact same as the tunnel-dest hosts bind port?

is this because the request handler has to make a tcp connection to it's own port on the server side? here's a full trace for 0.0.6 in a case where the local bind port is 8888 but the remote bind port is different:

[E 07:04:06.090 NotebookApp] Uncaught exception GET /http_over_websocket?min_version=0.0.1a3 (0.0.0.0)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/http_over_websocket?min_version=0.0.1a3', version='HTTP/1.1', remote_ip='0.0.0.0')
    Traceback (most recent call last):
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/web.py", line 1699, in _execute
        result = await result
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 278, in get
        await self.ws_connection.accept_connection(self)
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 881, in accept_connection
        await self._accept_connection(handler)
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 964, in _accept_connection
        await self._receive_frame_loop()
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 1118, in _receive_frame_loop
        await self._receive_frame()
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 1209, in _receive_frame
        await handled_future
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/ioloop.py", line 743, in _run_callback
        ret = callback()
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/websocket.py", line 658, in <lambda>
        self.stream.io_loop.add_future(result, lambda f: f.result())
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/gen.py", line 742, in run
        yielded = self.gen.throw(*exc_info)  # type: ignore
      File "/sandbox/.pex/install/jupyter_http_over_ws-0.0.6-py2.py3-none-any.whl.14c01a038a2557959603826dc297106055adf43c/jupyter_http_over_ws-0.0.6-py2.py3-none-any.whl/jupyter_http_over_ws/handlers.py", line 214, in on_message
        response = yield http_client.fetch(proxy_request, raise_error=False)
      File "/sandbox/.pex/install/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl.1ba9ff9792ecfed3e8b0a91ef2da93a1aaa27258/tornado-6.0.3-cp36-cp36m-linux_x86_64.whl/tornado/gen.py", line 735, in run
        value = future.result()
    ConnectionRefusedError: [Errno 111] Connection refused

kwlzn commented 4 years ago

friendly ping @colaboratory-team

vitchyr commented 4 years ago

@jiho

Also, I would welcome tips on how to allow connection without removing all security (no token, no password, etc.).

I'm not sure if this is what you meant, but your instructions worked for me even when I removed --NotebookApp.token='' --NotebookApp.password='', and used the URL with the token (e.g. http://localhost:8889/?token=autogenerated12345) from the remove machine. In other words

On the remote machine

jupyter notebook \ 
--NotebookApp.allow_origin='https://colab.research.google.com' \
--port=8889 \
--NotebookApp.disable_check_xsrf=True

Copy the generated http://localhost:8889/?token=autogenerated12345 from the remote server

On the local machine

ssh -N -L 8889:localhost:8889 user@server

Open colab on the local machine and use the http://localhost:8889/?token=autogenerated12345 URL from step 2.

rjmohit commented 3 years ago

I had the same problem having colab connect to my EC2 instance and the following snippet resolved it: jupyter notebook --no-browser --NotebookApp.allow_origin="https://colab.research.google.com" --ip=0.0.0.0 --port=8888 --NotebookApp.token='' --NotebookApp.password=''

googlecolab / jupyter_http_over_ws

Connection refused #2

tl;dr

Long version