Remote Python Development using Spyder

[abhijeetbadhe]

Hello,

I was wondering if it is possible to use the Spyder locally (on my mac) for development on remote Linux server with all the features (such as debugging, linting, variable-explorer etc.) available? All my project files would be on the server and the code would also run on the server...

From Spyder documentation and Carlos' posts on several forums, it's clear that it's possible to run a program developed locally on a remote machine (by launching spyder kernels, etc. ), but I have been unable to find whether or not remote development and execution is possible locally?

I would appreciate it if you could let me know...

Thanks & regards,

Abhijeet

[goanpeca]

Not at the moment but we do plan to add this capability for a future version.

[fohria]

I'm not sure I understand correctly what you want @abhijeetbadhe, but if you have SSH access to your server, you could mount the folder on your local machine using SSHFS: https://www.digitalocean.com/community/tutorials/how-to-use-sshfs-to-mount-remote-file-systems-over-ssh

I haven't done this with Spyder, as I mostly develop locally and sometimes connect to a remote kernel when I need some more computational power. But I've done it successfully with other editors in the past, and usually works like a charm!

[ccordoba12]

@fohria, we'd like to provide is a more integrated experience. So users would only need to provide their SSH credentials and then we'd create a remote kernel for them, show Files placed on the remote project dir and allow files to be edited in our editor.

[goanpeca]

Using something like this https://github.com/althonos/fs.sshfs @ccordoba12 ?

[ccordoba12]

Yep, seems like the right project to use.

[CAM-Gerlach]

@ccordoba12 Back when we discussed this at considerable length in the context of me writing a grant for Spyder 5 to "Army Labs" (by which I assume you meant either DARPA or ARL), with this as the headline feature, you found spurplus and recommended we use that, since it was a higher level solution than just SSHFS or paramiko alone. Might be worth digging that out if we want to implement something like this since we spent a lot of time discussing the high-level details of the implementation and how we would propose implementing it in stages.

By the way, after all the work we both put in on that grant, what happened to it? Did they have any feedback/suggestions for us about this remote development project? Did it even get submitted? Although I remember bugging you about it a lot, I don't recall if I ever got a clear answer; I know you were very busy at the time.

[ccordoba12]

you found spurplus and recommended we use that, since it was a higher level solution than just SSHFS or paramiko alone

Both projects seem similar. We'll have to evaluate which one is better maintained and more useful for us.

By the way, after all the work we both put in on that grant, what happened to it?

Sorry but support from TDK Micronas and Kite came in and it was much better than what we proposed to Army Labs. Besides, I couldn't make the contract with them through Quansight, which was a major issue for me. So I decided not to continue talking with them.

[CAM-Gerlach]

Both projects seem similar. We'll have to evaluate which one is better maintained and more useful for us.

Yeah, and I'm not sure fs.sshfs even existed back when we had that conversation for you to consider as an option.

So I decided not to continue talking with them.

Considering I put several straight days of my volunteer time into it at your request and you emphasized its urgency and importance to me at a time when I was very busy with other high-priority things, I would really have appreciated that at the time as a straight, honest answer to my many questions about it. But at the end of the day, that's almost a year in the past and we've both learned and grown a lot since then, so hopefully our communication has improved to a point where it won't be an issue in the future. I hope that grant can still be useful, and be submitted somewhere else at some point to get this funded.

[ccordoba12]

I would really have appreciated that at the time as a straight, honest answer to my many questions about it

Yeah, sorry about that. I was just in the middle of our initial talks with Kite and TDK people, so I forgot about it.

I hope that grant can still be useful, and be submitted somewhere else at some point to get this funded.

Well, basically all the ideas on how to improve our remote workflow capabilities came from it, so it was very useful just for that.

[fkloosterman]

Related, it may nice for Spyder to work with the Jupyter Enterprise Gateway to launch and connect to kernels remotely?

[ccordoba12]

@fkloosterman, very nice suggestion! We'll see what we can do about it.

[jsh9]

Just curious: RStudio Cloud is a browser-based RStudio -- the front end is elements (such as buttons, scroll bars, etc) that are rendered in the browser, and the back end is the R kernel on the server side.

Is it possible for Spyder to do something similar? I don't know enough about Spyder's front end to know the feasibility.

I know the scope of this issue is only limited to connecting to a remote back end, but I'm just wondering about this possibility for the future. :)

[ccordoba12]

No, it's not possible, sorry.

[CAM-Gerlach]

To expand a bit on @ccordoba12 's response, it is possible, but only in the sense that it would be a completely new frontend, sharing only the Spyder name (and perhaps some backend elements, but those are mostly lower in the stack anyway). It was possible for Rstudio because it was built on a web framework (just like VSCode is based on Electron), so it was relatively easy to port it to the web (but conversely, the Rstudio desktop interface does inherit some limitations as a consequence compared to a "true" desktop application.

While some of the panes (Help, Online Help, and the future Viewer pane) do leverage web-based rendering, Spyder as a whole is written entirely in pure Python, and the UI uses PyQt (a Qt binding for Python), powerful, full-featured desktop GUI application development library. This offers considerable advantages of performance, OS integration, and a native experience users expect, alongside being written in the same language it is primarily used to develop to make it easy for users to contribute features and write plugins, but isn't as readily portable to the web.

However, if funding or community support is available, there have been plans to considerably expand Spyder's remote development features, including potential integration with Jupyter. Perhaps @ccordoba12 can comment on the current status of that; we've been talking about it for years but money and developer time has always been a limiting factor compared to far more lavishly funded projects like Jupyter.

[jsh9]

Thanks for both your answers @ccordoba12 @CAM-Gerlach .

I didn't know that Jupyter is better funded than Spyder. If I had known, I probably would have gone there and ask them: "can Jupyter implement a good variable explorer and a proper debugger, just like what they do in Spyder?" I'm surprised that they still don't (have a variable explorer or a debugger), after this many years.

I would say the scope of this issue (i.e., kernel on the server side + front end on the client side) covers 95% of our use cases, and that's quite enough. (The only remaining 5% use cases where we desire a full browser-based IDE is that we somehow want to code on tablets.)

[CAM-Gerlach]

They get $100 000s or millions from large companies and foundations, I'm not entirely certain where all that money goes, vs. Spyder is developed and in large part crowdfunded by the international community of its users, although it has attracted a rapidly increasing amount of grant money and corporate donations in the past couple years, mostly to implement the most-requested features by users (the two top ones which were better code completion/introspection/analysis and a better debugger, which were the two biggest major enhancements delivered in Spyder 4).

I'm surprised that they still don't (have a variable explorer or a debugger), after this many years.

Yeah, I'm not sure, considering even VSCode is getting at least basic support for that sort of thing now. It does help that Spyder is more mature than e.g. JupyterLab and written in Python, attracting a ready pool of talent and avoiding some of the hurdles of web-based development.

[ccordoba12]

To expand a bit on @ccordoba12 's response, it is possible

I'd like to clarify that this is possible in theory, but we haven't considered to follow that direction at all. So, in practical terms, you're not going to see a purely web-based Spyder in the future.

However, Spyder can run on Binder. That's the closest you'll get to "have Spyder inside my browser" from us.

[texadactyl]

Personally, I routinely connect over SSH to a data centre. The files that I sometimes have to test with are in the 5-400 GB range (telescope data). Most of the new software makes use of the NVIDIA GPU resources for several different functions but spyder need not be concerned with that.

I can easily live with debugging via print and logging.debug statements as I have for a long time. But, it would be highly productive to be able to use spyder edit & code analysis on a set of Python source files as opposed to individually vi/nano and manual execution of pylint. I could even edit my configuration files, bash scripts, and C source.

You might ask, why not use an existing facility?

• Visual Studio is a confusing user experience for most scientists.
• Virtual desktops are a confusing user experience for most scientists. They also might be impractical, depending on the user.
• Jupyter is an optimal look-and-feel experience but its functions are limited.

[CAM-Gerlach]

I know at one point @ccordoba12 and I discussed this at length, and following that I actually wrote a grant proposal to get a set of major enhancements related to this funded for Spyder 5, as one of if not the headline new feature. @ccordoba12 had apparently found an agency interested in funding it and I managed to turn a pretty solid proposal around on a pretty short deadline, but unfortunately, for reasons that are still unclear, someone never actually sent along the proposal, so it never happened, and the funding we eventually got for Spyder 5 went in a very different direction instead (the plugin system).

However, its possible that the plugin API could be used to develop a plugin that brings better support for this to Spyder. Of the ways I remember doing it, IIRC it came down to either something like SSFS or interfacing with a Jupyter server running on the host to provide the filesystem services we needed.

@texadactyl Have you tried the existing SSHFS + connect to remote kernel approach? If you're already connecting over SSH, it should definitely be possible. I'm not 100% sure that runfile will work depending on how Spyder does the file paths, but run selection, run cell (if Copy full cell contents to the console is turned on under Preferences --> Editor --> Run Code), etc. should work since its just sending the text from the file to the console, and the various other panes should work. Heck, you can even just use Spyder for everything but the console stuff and then use console commands or the Spyder-Terminal first-party plugin to execute runfile or run your full scripts/etc. in another Python interpreter, without leaving Spyder.

[texadactyl]

@CAM-Gerlach I am sure that I and the other software engineers (mostly undergrads) could figure out something for ourselves. However, the scientists are not going to go for this. I was hoping for something as simple as what Visual Studio or Jupyter provides:

• Install a plugin.
• Point the plugin at the SSH configuration file customised for the destination.

This was already difficult to set up & maintain for 40+ scientists who are highly intelligent but want to remain focused on Physics. Cannot blame them.

[CAM-Gerlach]

@ccordoba12 Looks like there are at least a couple SSHFS/like backends (e.g. sshfs, fs.sshfs, etc) package on Conda-Forge that could be used for this. How feasible would it be to create a plugin using the new API that would allow, e.g., connecting to, browsing and opening files from them via the Files/Projects pane? We already use paramiko for ssh connections to remote kernels and it sounds like one of them is based directly on it, so such a plugin could reuse a lot of the existing infra. The other component would be the plugin ensuring that the correct (remote) paths are sent to the console; its simple in theory but not sure how hard it would be with the new API.

I'm not sure how available someone on the core team might be to work on this, but one of the volunteer core devs might be interested, we could at least help you along, and there's also the community work order process with Quansight; I don't think it would take that much funding to do. I'd also be interested in this myself (though my time is stretched very thing as it is), and I encourage committing some of my OC donations over the years toward this.

[ccordoba12]

@texadactyl, thanks for the feedback. I'd like to say that this is really top in our priority list, we asked for funding this year and didn't get, but we are expecting to get it from another source next year. Hopefully we'll have an initial solution in a year or so.

[raphaelchinchilla]

@ccordoba12 I read somewhere in Spyder's website (cannot find it again) that you all got funding to implement this functionality. Any EDA?

Edit: I found where I read it

[ccordoba12]

@raphaelchinchilla, the first part of that project (i.e. a much simplified and friendlier way to start consoles in remote servers) will be released in Spyder 6, to be released at the beginning of next year.

[texadactyl]

@ccordoba12 When you get into design, please consider that you will have several kinds of users with varying levels of experience with SSH security. Are you assuming that the user is solely responsible for the secure connection from the user's computer to the server that is intended for use? That is the simplest/safest from a Spyder point of view. Spyder doesn't get involved in the administration of the user's security and networking. Keep in mind that it is common to be employing multiforwarding (multiple hops through proxies) using certificates.

[ccordoba12]

Are you assuming that the user is solely responsible for the secure connection from the user's computer to the server that is intended for use?

Yep, we are.

Keep in mind that it is common to be employing multiforwarding (multiple hops through proxies) using certificates.

Ok, thanks for letting us know about it. Do you know how we could set up something like that so we can test it?

[texadactyl]

I adapted this from a real-world set up that is in heavy use. Please try it out or the 3-node shortcut (no middleA node).

Assumption: Using standard SSH protocol with RSA certificates (identity file). 
Nobody likes manual login from hop to hop although that is 
done the first time as you are generating certificates.

Here is a typical model with 4 computers, 3 in a data centre:

   User PC -----> publoginA -----> middleA -----> pydest1
   Home           Internet         private        private
                  accessible       traffic        Python
                  (froggy.org)     monitor        development
                                                  environment

Reference: https://linuxize.com/post/how-to-setup-passwordless-ssh-login/
One can follow that script for:
* User PC to publoginA (https://www.froggy.org)
* publoginA to middleA (not Internet accessible)
* middleA to pydest1 (not Internet accessible)

Yes, you can adequately test without middleA. If doing that, adjust the configuration file.

SSH Configuration
=================

Host pydest1
  ProxyCommand ssh middleA -W %h:%p
  IdentityFile /home/frog/hopper/middleA.id_rsa
  User frog
  ForwardX11Trusted yes
Host middleA
  ProxyCommand ssh publoginA -W %h:%p
  IdentityFile /home/frog/hopper/publoginA.id_rsa
  User frog
  ForwardX11Trusted yes
Host publoginA
  HostName froggy.org
  User frog
  IdentityFile /home/frog/hopper/froggy.id_rsa
  ForwardX11Trusted yes

Connection Script
=================

USERID=frog
TARGET=pydest1
ssh -Y -F configuration-file-path ${USERID}@${TARGET}

[raphaelchinchilla]

Hi all, I understand very little of the protocols itself. However, there is something called mosh, the mobile shell https://mosh.org/, that allows the connection to the server to drop and reconnect without losing progress. If that could be integrated into Spyder... No other IDE has that

[jsh9]

Hi @ccordoba12 , does your team have access to AWS SageMaker? (Personal/non-profit/education accounts may be cheap-ish.) More and more people are using SageMaker to develop/train machine learning models, so it may be good to test Spyder's remote development on SageMaker. (They currently offer "SageMaker Studio", which is just glorified Jupyter Notebook. And people are asking for "full-fledged IDEs" in SageMaker.)

[ccordoba12]

@jsh9, we'll try to cover that use case too, i.e. connecting to a remote JupyterHub instance from a local Spyder installation to run computations on it. But that's not part of the plan of this grant.