anaconda mode lost rpc connection after some time

ivonwrk commented 4 years ago

Hi. I'm constantly getting the same error: after some time (tens of minutes), anaconda mode could not get the response. If I kill the buffer anaconda-mode, and after that all works fine.

Below is status of buffers after failure: anaconda-mode anaconda_mode port 36909 127.0.0.1 - - [05/Dec/2019 15:01:08] ================================================================================ Traceback (most recent call last): File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 101, in apply result = method(params) File "", line 98, in wrapper result = f(jedi.Script(source, line, column, path, environment=virtual_environment)) File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/jedi/api/init.py", line 118, in init cache_path=settings.cache_directory, File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/jedi/evaluate/init.py", line 388, in parse_and_get_code return self.grammar.parse(code=code, path=path, kwargs), code File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/grammar.py", line 76, in parse return self._parse(code=code, **kwargs) File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/grammar.py", line 126, in _parse new_lines=lines File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/diff.py", line 152, in update self._copy_from_old_parser(line_offset, i2, j2) File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/diff.py", line 211, in _copy_from_oldparser from = copied_nodes[0].get_start_pos_of_prefix()[0] + line_offset File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/tree.py", line 264, in get_start_pos_of_prefix return self.children[0].get_start_pos_of_prefix() File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/python/tree.py", line 125, in get_start_pos_of_prefix previous_leaf = self.get_previous_leaf() File "/home/oleg/.conda/envs/pytorch/lib/python3.7/site-packages/parso/tree.py", line 74, in get_previous_leaf i = c.index(node) ValueError: <Function: create_dirs@1168-1179> is not in list

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/providers/basehttp.py", line 40, in do_POST status, response = self.server.service(data) File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 50, in call result = self.apply(method, args) File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 105, in apply server_error(args['id'], error) File "/home/oleg/.config/emacs/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/errors.py", line 91, in server_error raise ServiceException(500, dumps(response)) service_factory.exceptions.ServiceException: (500, '{"jsonrpc": "2.0", "id": 1, "error": {"code": -32000, "message": "Server error", "data": "ValueError(\'<Function: create_dirs@1168-1179> is not in list\')"}}')

TerryTsao commented 4 years ago

I'm getting very similar errors recently

CeleritasCelery commented 4 years ago

Can you see if reverting to commit 3ede03d40288079a078f5aa6f8e26eeffe2d6962 fixes the issue?

cc @falloutphil

falloutphil commented 4 years ago

Hi,

I'm the author of https://github.com/pythonic-emacs/anaconda-mode/commit/1bc301b2d2bc336988f4a16a891c275a90136ca5 which was recently reintroduced.

I've just tested doing method look-ups on TRAMP over an approx 25 min period on a version containing https://github.com/pythonic-emacs/anaconda-mode/commit/1bc301b2d2bc336988f4a16a891c275a90136ca5, but without other recent changes.

Things seemed fine, but my use-case and setup could be different to others. So I'll add a bit more explanation.

My commit does 3 things - all of which are to do with the underlying creation of an ssh tunnel as a separate process for RPC calls to travel through:

Most importantly - fixes a bug that means that the original unsecured RPC connection created which completely ignores that ssh tunnel is replaced with a secured RPC connection through the ssh tunnel. By coincidence the original code would work in some cases, but not by design and by passing cleartext over the web.
Secures tunnel creation by invoking tunnel on remote localhost, rather than pythonic-remote-host, this is important to make sure only the creator can access the tunnel, rather than third parties.
Introduces an optional "-J" jump proxy switch, so multihop TRAMP connections are seamlessly supported.

If you're not using TRAMP then it's hard to see how my commit would have any effect.

If you are using TRAMP, I have a theory!

You're one of the users where the old code worked by chance, and it was simply ignoring the ssh tunnel it was meant to be using!
But now that I've fixed the code the ssh tunnel is being used.
Some ssh servers have a timeout, so idle tunnels could be disconnected after a while.

This scenario would match the reported symptoms, so although I'm making a pretty wild guess, it sounds plausible to me.

So if my theory is correct - all you need to do is add a keep alive to your ~/.ssh/config file - Google will help here, but something like the below should work:

Host *
     ServerAliveInterval 300
     ServerAliveCountMax 2

Note I don't want to make the keep alive part of anaconda-mode itself as this is too intrusive. The end user should configure this as required.

Also note as I've mentioned previously you also need to make sure that each remote host and proxy jump used must have been previously added to the known_hosts file. Although this isn't the issue in the case explained as nothing would work for any time period if this was the issue.

If this doesn't help, then please can you reply with more details of exactly how to recreate the problem - some helpful questions:

Are you opening a remote python file using TRAMP?
Does the situation exist if you only use anaconda-mode on local python files?
If using TRAMP, are you proxying the connection via a third or fourth server or connecting directly from your local emacs to remote server?
You're then editing a file, but after some time Anaconda functions are not working (could you give an example of something that works and then fails)?
Are you editing files on more that one remote server at the same time?
Can you post the contents of both anaconda-mode and anaconda-ssh buffers
What versions of python, jedi, and service-factory do you have installed locally/remotely.
What does "ps -ef | grep ssh" return on the local server?
Are you using a virtual environment, and if yes, is the situation exist if you don't use a virtual environment?

Sorry for the long message - hopefully it helps!

CeleritasCelery commented 4 years ago

This line does not seem related to tramp. Why was it changed?

https://github.com/pythonic-emacs/anaconda-mode/commit/1bc301b2d2bc336988f4a16a891c275a90136ca5#diff-d4bac877ea8ba1b9051ada5ac9a964baR502

falloutphil commented 4 years ago

This line does not seem related to tramp. Why was it changed?

1bc301b#diff-d4bac877ea8ba1b9051ada5ac9a964baR502

After creating the local tunnel you want the RPC requests to go down the tunnel, not direct to the host. The original code was completely disregarding the tunnel it had created, which I believe was a bug - it was working sometimes by fluke (when the randomly generated RPC port happened to be open on both servers), but was sending data as cleartext, which is a bad idea.

This change ensures RPC requests go down the tunnel (on port 22) and pops-out on the server where the TRAMP file is hosted. This avoids issues with firewalls blocking direct RPC requests as well being secure.

CeleritasCelery commented 4 years ago

Closed with #379

pythonic-emacs / anaconda-mode

anaconda mode lost rpc connection after some time #376