RFE: Support Remote Interpreter

[DonJayamanne]

From @satyenr on May 13, 2016 3:37

The ability to run code on a remote machine is awesome. You can develop on a Windows PC or a Mac while running and debugging on a remote Linux server. PyCharm supports such a feature - https://www.jetbrains.com/help/pycharm/2016.1/configuring-remote-python-interpreters.html.

Having a remote interpreter also helps when the remote machine already has a functional Python environment - possibly managed by someone else - which you don't want to/can't replicate on your local machine.

Copied from original issue: DonJayamanne/pythonVSCode#123

[DonJayamanne]

This is a great suggestion. The architecture of VSCode and the extension supports this. However this would require a lot of effort, primarily in two parts. One would be to sync the two environments and the other would be the creation of a proxy component that sits on the server to manage all of the comms and surely enough this would have to be developed in python.

However, this would require weeks if not longer to build. But something that I will most definitely explore as a side project.

[DonJayamanne]

From @satyenr on May 18, 2016 11:51

Thanks for accepting this RFE. I was trying to add this feature myself, but the lack of knowledge of VSCode related stuff and other priorities forced me to put that project aside for the time being.

[DonJayamanne]

Todo: Add support for docker containers as per #179

[DonJayamanne]

Suggestions for #183

[DonJayamanne]

@vbem, @satyenr , @satyenr, @nffdiogosilva Hi everyone, I would like to re-visit this to understand the requirements and the constraints.

Here are some of the environments (usages):

• Docker/vagrant containers
• Remove servers (linux, mac, windows)

Requirements:

• Linting (code analysis)
• Code navigation (find references, go to definition, etc)
• Refactoring
• Debugging (done, this works today, but SSH is to be looked at - separate issue)

One possible solution I'm thinking of is:

• Install nodejs on the remote server
• Install a npm package (nodejs module)
• Start a socket server on this target server
• VS Code will communicate with the remote server over a socket connection

This is the easiest solution and quickest to implement. However, this requires the installation of:

• nodejs on the server
• an npm package

Would this be acceptable?

The only other solution is the following one. However this would take a lot longer to develop and implment:

• install a python package on the server

[DonJayamanne]

From @satyenr on October 4, 2016 9:12

When I logged the ticket, I had the following use case in mind:

• Remote server has a fully configured Python environment, which is more or less identical to the deployment setup.
• The developer workstation may have Python installed, but it doesn't have the fully configured environment. While it is possible to build a virtual environment containing the required modules in many cases, some modules don't work cross-platform, or are hard to compile. In any case, it is always preferable to development using an environment that is similar, if not identical, to the deployment environment.
• The development and deployment environments may even be running different operating systems.

As a result, linting, code-completion etc don't work as expected and end up causing more trouble than helping. A workaround is to run VSCode in a VNC session, or using X forwarding. In both cases, responsiveness of the UI is a function of network connectivity/load.

Originally, I had imagined that it will be implemented by running commands on the remote machine over SSH - that is what PyCharm does. Not sure if they maintain the connection as long as the IDE is open.

Using SSH ensures that you don't have to worry about things like authentication, managing a daemon on the remote machine etc. But using SSH means assuming that the client has a working SSH client - something you can't assume about Windows clients. I suspect PyCharm may be using its own SSH client - something I don't think is trivial to implement.

Your idea of using a socket server on remote machine seems like a good one. I don't particularly care for which language it is implemented in. Python would be ideal, since we are anyway assuming that python is installed on the remote machine. But I am fine with NodeJS as well. As long as the protocol is open, someone can always implement a python version later.

install a python package on the server

I assume you meant - install a python package on the remote which is then run as a daemon listening for requests from VSCode. Let me know if that is not the case.

[DonJayamanne]

@satyenr

I assume you meant - install a python package on the remote which is then run as a daemon listening for requests from VSCode. Let me know if that is not the case.

Yes, you are right. Will try to start with the node daemon, and later move it to a python daemon.

Finally, an open protocol does exist today: https://github.com/Microsoft/language-server-protocol

Again, implementing this will be done at a later stage.

• Phase 1: Custom protocol with solution developed in nodejs
• Phase 2: microsoft language server protocol
• Phase 3: Develop this in Python

[DonJayamanne]

From @satyenr on October 4, 2016 9:47

Sounds like a good plan to me. I can help with the Python part if you want.

I don't have any experience with NodeJS, unfortunately.

[DonJayamanne]

From @nffdiogosilvadc on November 11, 2016 23:50

Hello =),

How is the current state of this issue?

Thank you

[DonJayamanne]

@nffdiogosilvadc no work has started on this, as there isn't much interest in this feature. Except possibly for the two of you.

[DonJayamanne]

From @aventurella on November 28, 2016 14:24

I'll raise my hand for this. Our whole dev team formerly relied on vagarant and currently relies on docker to do all of our development. Specifically during development, we use docker-compose to ensure all of the devs are running the same environment.

Today we use Anaconda + Sublime. I have been exploring Visual Studio Code for Python development and something similar is a must. This looks to be where it will originate from if it is to be. I can't thank you enough for even discussing it.

In our docker development case we just start up a new container that has the anaconda remote server on it. It gets the volumes from the django application, so it has access to all of the code currently running:

volumes:
  - ./django:/usr/app
  - /usr/local/lib/python3.5

We also deploy on docker, so if this were a: We need to debug a running server we would end up doing something similar, just attaching the container and disposing of it when done. From that point of view I have absolutely no issue with this workflow:

- Install nodejs on the remote server
- Install a npm package (nodejs module)
- Start a socket server on this target server

I am also aware that not everyone uses Docker. I only mention the Docker use case here for those that may also depend on it. Naturally every deployment would differ and have different needs.

[DonJayamanne]

From @mgallagher on December 3, 2016 5:2

+1 this would be amazing

[DonJayamanne]

From @nffdiogosilva on December 7, 2016 15:26

Now that it seems that more people are interested in this feature, will it return to roadmap again?

[DonJayamanne]

Agreed, looks like there's enough interest here to get this into the roadmap now. There are two ways to go about this. I'll post both options, so as to get the feedback.

[DonJayamanne]

There are two ways to implement this feature:

Option 1: Add dependency on node.js on the remote server

Benefit: Easier to develop This is the easiest option for me (less python code), more node.js (javascript) code - i'm familiar with this

[DonJayamanne]

There are two ways to implement this feature:

Option 2: Add dependency on node.js on the remote server

Benefit: No dependency on node.js (except some other python libraries that I'll have to write) Will require more work as I'll need to write everything from scratch using Python (this isn't my language of choice).

[DonJayamanne]

FYI, I'll commence the work with the node.js approach (Option 1).

[DonJayamanne]

From @satyenr on December 8, 2016 3:47

Option 2: Add dependency on node.js on the remote server

You meant, "Add a Python module on the remote server", methinks.

[DonJayamanne]

From @sierran on December 29, 2016 12:25

For my team it is mandatory functionality without which we can not work, so fingers crossed :)

[DonJayamanne]

From @ianfitzpatrick on January 3, 2017 5:47

Hoping for this too. This is single feature is what's keeping me from switching to Visual Studio Code. Would be amazing if this lands!

[DonJayamanne]

From @wasinski on January 17, 2017 16:50

Another word of encouragement: this will be an absolute killer feature!

[DonJayamanne]

From @peymanmortazavi on January 20, 2017 23:46

👍 I'll switch to VSCode at work if this feature becomes available.

[DonJayamanne]

From @gulshan on January 25, 2017 9:55

https://github.com/palantir/python-language-server Will this be any help?

[DonJayamanne]

Thanks, that just might work

[DonJayamanne]

From @adlermedrado on February 13, 2017 16:2

+1 to this feature. It'll be great.

[DonJayamanne]

From @satyenr on February 13, 2017 16:32

Seems like Anaconda (Sublime Text) has implemented this feature using a similar approach. A daemon (written in Python) listens on a given TCP port in the remote host. I have tried it - and my initial impression is that it works well enough.

@DonJayamanne If you haven't started working on this already, it may be worth looking at Anaconda to see if the server can be reused here. The code is available under GPLv3.

For what it is worth, PyCharm maintains an SSH connection with the remote host to achieve this.

[DonJayamanne]

From @YAmikep on March 10, 2017 18:12

👍 Hoping for this too. Would be amazing if this lands! Thanks

[DonJayamanne]

From @liqing-ustc on March 20, 2017 5:20

Hoping for this feature!!!

[DonJayamanne]

From @tony-garcia on March 23, 2017 19:31

+1

I'm using VSCode for most things these days but for a few projects the dev server is in a python Docker container so I'm still using PyCharm for those (since it allows associating a project to a python interpreter in a Docker container or docker-compose service). Would be cool to just use VSCode for everthing.

[DonJayamanne]

From @johnnywell on April 2, 2017 2:13

+1 I recently heard about this extension on Talk Python to me and switched to VS Code since then. I'm really looking forward to seeing this in action!

[DonJayamanne]

From @jaymickey on April 8, 2017 5:43

+1. I develop mostly on a Windows machine, and had planned to utilise a Linux Docker container to develop a flask based web application, but was disappointed to see that there is currently no way to do this while still having access to code completion (intellisense), linting, etc.

[DonJayamanne]

From @wsgeek12 on May 5, 2017 19:50

This would be great yes yes yes +1

[DonJayamanne]

From @IBestuzhev on May 11, 2017 9:31

+1. It would be a nice feature as I often use Docker (and sometimes Vagrant) to set-up the environment.

I can share what I use now.

For pylint it's quite easy, but does not work too fast unfortunately. I used a custom path for pylint and point it to my custom .pylint.py (gist).

I was trying to do the same with jedi, but I did not manage to pass the stdin/stdout connection. I modified python.jediPath to point to jedi.py file from that gist. Unfortunately due to some stdin/stdout issues I was able to do it only if I modify completion.py (the one in the docker) to process one request per time. I just added a break to watch method. Also there is something in docker that prevents from running it under exec, only run works.

This jedi setup is working but it's really slow to use. Obviously it's a terrible idea to deploy the container every time, it was just proof of concept. I think replacing the communication to tcp will speed it up.

So ideally is to have TCP communication between client/providers/jediProxy and pythonFiles/completion. In this case it will be easy to port-forward the connection to remote host.

Or as workaround it can be TCP communication between jedi.py from python.jediPath to modified completion.py that runs on remote. But this way is dirtier, as custom jedi.py file should block on import and prevent execution of some code from original completion.py.

Another thing that is needed here is path mapping.

But I hope someone may find this ideas useful and can extend it.

[DonJayamanne]

@IBestuzhev wow, thanks, will try to get started on this. busy with a lot of things. But this will surely make things easier, at least for docker.

[evgkirov]

+1 for docker support

[DonJayamanne]

@kirov please up vote the topmost comment, that's makes it easier to count the number of votes on the feature request.

[arianamiri]

+1 for both docker and general remote interpreter support

[brettcannon]

Just an FYI for anyone who leaves a +1 comment: we are only paying attention to the 👍 / 👎 reactions on the top comment and not the amount of comments left as it's easier to track and it doesn't generate an email for everyone who happens to be following this issue for updates (which anyone can do by subscribing to updates; there's a button in the left-hand column for that in GitHub's web UI).

[mrkafk]

Not having this feature is a serious problem if your local machine and remote environment vary a lot. E.g. support of Pytorch for Windows is still not complete and I have to develop on Windows (corpo requirement) while remote environment is Linux.

[valignatev]

If you don't mind, I'll copy my comment from the original issue about sshfs workaround I'm kina using now:

While we all patiently (or not so patiently ;)) waiting for this feature to be implemented, I'll describe sshfs workaround I use now: I have dev docker container with python and all site-packages installed and opened ssh on some port PORT. So I've installed sshfs (little guide for ubuntu can be found here), and then I've mounted my docker container's fs to my local host like this:

sshfs -p PORT -o idmap=user docker_ssh_user@docker_host_or_ip:/ ~/.docker/mountpoint

And now I'm able to select workspace interpreter from inside vscode or just paste its path to .vscode inside my project root. Something like this could work:

"python.pythonPath": "~/.docker/mountpoint/usr/local/bin/python3.6"

Hope it'll help somebody.

[balta2ar]

@valentjedi What about performance or any other issues? Do you notice anything unusual or has it been a very "smooth sailing" so far?

[valignatev]

It is pretty smooth for me for now. My main use case is local docker daemon with ssh for local development, so for me, it's really fast. I haven't tried to mount actual ssh FS from some really remote server. But I hope that there is some caching to speed things up.

[williamstar]

extrmely desire support for remote deploy !!!

[dan-kez]

@valentjedi I'm working on porting your idea for use with docker. I originally was globally installing my packages but I think I'll use a virtual env and mount that directory. That way I can apply the interpreter locally.

Has anyone else done this?

[IBestuzhev]

@dmk255 I did similar thing, but with Bash on Windows. I had project installed in Python under Bash, but folder is visible from host system. And so I added this folder to python.autoComplete.extraPaths. But you need to have the same version of Python installed both on host and remote.

[alyoshark]

Hi @valentjedi , the idea sounds pretty cool but was wondering if you had any issue if there are runtime library differences, for example if the python container is from an Alpine image and the hosting server is Ubuntu, Mac or even Windows

[valignatev]

Hi, @xch91 This would totally be an issue if you try to actually run the code from within mounted partition with your local python. And I've experienced it with, for example, MySQL libraries for python. But if you'll use it only to get goto definition and completion/other Jedi stuff, it should be fine.

[alyoshark]

@valentjedi noted with thanks. A more full-fledge debugging environment is more desirable but basic code intellisense is surely a better experience already.

[dArignac]

I'd also like to see this realized as our teams wants to switch from PyCharm to VisualStudioCode. We run the IDE on Windows and our software is running in Docker containers in Vagrant Box.

[b99andla]

+1 from us as well, it would be the final step holding us back from switching over from PyCharm.