Elpy in Tramp buffers

[jorgenschaefer]

Elpy currently does not work well in tramp (remote) buffers. As @UnwashedMeme summarized in #170:

Elpy currently only runs python processes locally(1). When you are working on a remote tramp buffer it tries to start the elpy-rpc backend on the localhost with a default-directory that might be remote (not exactly sure how that's working) with the following observed behavior for elpy-rpc-backend:

• "jedi" seems to do alright, but offers completions based on locally installed packages, not what's on the remote
• "rope" chokes with the error of the ticket title
• "native" works fine but has the limitations of native.

• nil might detect rope on the localhost and err.

(1) At a minimum, to make this more tramp aware, https://github.com/jorgenschaefer/elpy/blob/master/elpy.el#L2274 needs to be changed to start-file-process and have the libraries installed remotely.

I am not sure what the correct behavior for Elpy would be. It can be one of two:

1) Elpy treats the remote buffer like a local buffer, using a local Python installation and local Python modules. This is tricky, because Elpy would need to find a remote project root, and backends likely dislike that (that's what Rope is annoyed at, I think). I'm not sure at all what the correct behavior here would be.

2) Elpy uses start-file-proecess as suggested. Then the helper functions will have to deal with remote buffers, telling the user that remote buffers require an Elpy installation on the remote system. A lot of complexity for a niche use case.

I'm not sure what the correct behavior should be, so I'll leave it at the current (suboptimal) behavior for now. If you have suggestions on how this should be handled, please do not hesitate to comment.

[UnwashedMeme]

What's correct? My vote would be option 2 above: try and deal with the remote python system. That is a lot harder though. As it stands right now if you set the rpc backend to jedi you get option 1.

I have a lot of python scripts for managing servers and in more cases I'm running tests and experiments in containers via ssh+tramp so it feels less like a niche to me ;-) Not so much that I'm going to have the complete pull request for you shortly though. oh well.

---

A bit more on what I found: When rpc is launched with start-file-process it does correctly try to launch python on the remote system. From different starting points this is what it looks like:

• nothing installed remotely: Tries to launch remote process (and gives the installation warning) repeatedly (every keystroke or so).
• elpy installed remotely: no backends: gives the installation instructions once. elpy process remains running remotely.
• rope installed remotely: Error from RPC: "'NoneType' object has no attribute 'pycore'" as in #170
• jedi installed remotely: appears to work gives completions based on packages installed remotely that aren't installed locally.
• ipython installed remotely: appears to work socket.gethostname() is the correct host

So things to do include:

• Handle elpy not being installed at all more gracefully (this might also be good for local installations)
  • Auto fallback to native?
• rope: debug whatever its problems are (or only use jedi for remote; show warning/instructions if only rope found on a remote connection)
• tests (I haven't been using these; just looked at the work in 140 and 181; looks interesting but doesn't appear to be in release yet)
• pyvenv?

I'm sure there's more as I don't know every part of this wonderful library.

[jorgenschaefer]

Just to clarify: I'd love if it Elpy would work correctly for tramp users. My problem with that is that I have never managed to have tramp work correctly for me outside of the trivial case ("edit file remotely, nothing else"), which makes it kind of difficult to understand what Tramp really wants or what the "correct" behavior for tramp would be (outside of "it should do what I want").

Rope's problem is that the project root it is told about does not look like a real file. It gets "/ssh:foo/bar/baz", and that doesn't exist on either system. Elpy would need to translate that path to a local path for the remote process in a tramp buffer.

Likewise, Jedi will add the /ssh:../ directory to sys.path, which does not break anything, but if that directory is not in sys.path already,

pyvenv simply sets environment variables. I have no idea how tramp translates the local exec-path to the remote system, and pyvenv can't really set both … I'm not sure there is a good solution for using virtualenvs remotely.

If you want to be a bit adventurous, could you try this with the current unstable Elpy from the master branch? (git checkout and adding the git repo to load-path and PYTHONPATH should be enough) Elpy 1.5 change the error handling a lot.

(I'm not usually in the business of telling others how to do their work, so take this just as an idea, nothing more, but have you considered using sshfs to edit files in your containers locally? This means having a local virtualenv with the appropriate libraries, but it has been a lot more stable for me than tramp's attempts at working locally and remotely.)

[bwo]

I don't think this is niche---I'm developing right now in a vagrant box where I have lots of non-python dev dependencies, and elpy really does not like this. I've kind of tricked it in some respects by setting the python rpc command to be a local shell script that sshs into the box and loads my venv and executes python there, but for lots of features (test running, etc.) it's a PITA as it stands.

[jorgenschaefer]

I am sorry this is a problem for you. Tramp uses a very serious amount of hacks and kludges to make things work like this, and it is not trivial for Elpy to add another layer of hacks on top of tramp to make this work. As I do not use this kind of setup, it is difficult for me to fix. If you feel this is an important feature, please do implement it and provide a pull request! Thank you :-)