Remote container with Singularity

[statiksof]

Is it possible to use singularity containers instead of Docker? We only provide Singularity on our remote machines.

[Chuxel]

Not today. Only Docker is fully supported today. Kubernetes has attach support and Podman works with some limitations.

[statiksof]

@Chuxel thanks for the quick answer. Actually, I though that I can overcome the glibc issue by using containers. Is there any other workaround to make the remote ssh works? I have glibc 2.12.

[Chuxel]

Not today - Node, V8, and even Chrome require recent versions of glibc unfortunately.

[statiksof]

Just an update, I can start the vscode-server inside the singularity container. It starts without any issue. The problem now is how to forward ssh or config the ssh extension locally to use this server.

[oschulz]

I would love to see support for Singularity containers, together with other container systems like Shifter and Charliecloud that are popular in scientific (resp. HPC/HTC) computing environments.

I currently use a very clunky workaround:

• I use a custom SSH script (via VSCode config option"remote.SSH.path") that sets an env var LC__SINGULARITY_IMAGE (different images can be chosen depending on the name of the target system) and then does exec ssh "$@". Env vars starting with LC_* (and only those) are usually forwarded by SSH.

• I modify "$HOME/.vscode-server/bin/*/server.sh", and add change "$ROOT/node ..." to singularity run "$LC__SINGULARITY_IMAGE" $ROOT/node ..."

It works, but it's not very convenient. "server.sh" needs to be modified after every update of vscode-server, and since the server instance gets reused, only one container instance can be active on a given remote system at the same time.

[oschulz]

A first step to make it easier for users to use custom container runtimes might be to add a configuration option like "remote.SSH.runwith". It would allow the user to specify a wrapper script for the vscode server. That user-specified wrapper script would be required to end with exec $@, so that "remote.SSH.runwith" = "run_in_my_container.sh" would result in run_in_my_container.sh "$ROOT/node" ${INSPECT:-} "$ROOT/out/vs/server/main.js" "$@" or run_in_my_container.sh /path/to/server.sh ....

If VS Code would also set some env vars before calling that wrapper script, forwarding information like workspace path, then the wrapper script could decide which container image to run, and how, depending on the application.

[statiksof]

@oschulz thank you for sharing your workaround. I finally got it to work after multiple refinements. For me, forwarding the LC* didn't work. So, I spent most of the time modifying server.sh. I used a default centos 7 image just for testing, there are some warnings when the agent starts, I am thinking to provide dedicated images for that purpose.

[oschulz]

For me, forwarding the LC* didn't work.

Unfortunately, that depends on the SSH server config - I guess some sites disable it, and users don't have control over the server config, of course.

[RobertRosca]

The instructions for using Podman with the remote containers are here: https://code.visualstudio.com/docs/remote/containers#_can-i-use-podman-instead-of-docker

It just suggests pointing the docker executable path to the podman path, and it works as podman and docker have similar CLI commands.

One solution would be to create a wrapper for Singularity which parses and 'translates' the docker commands into Singularity commands. This is similar to the workaround mentioned here int this comment above https://github.com/microsoft/vscode-remote-release/issues/3066#issuecomment-698816982 but it's slightly less clunky.

I'll add this to my list of cool things to maybe work on, but probably won't get to it. If anybody here is interested I'm willing to help out though.

[oschulz]

Thanks @RobertRosca , I'll definitely try that, such a wrapper should be possible. Will post if it works.

[david-macleod]

Another workaround (https://github.com/microsoft/vscode-remote-release/issues/4474) using a modified version of the remote extension (which will hopefully make it into the default extension soon) allows specifying the singularity exec in .ssh/config e.g.

Host blah
    User blah
    HostName blah
    RemoteCommand "/usr/local/bin/singularity exec $CONTAINER_IMAGE /bin/bash"

[oschulz]

I guess the ideal solution would be a config option for remote-ssh that allows the user to specify an arbitrary command that gets inserted before "$ROOT/node [...]. That way, one could use workspace-dependent containers, etc.

Was trying to prototype something like that, but to my knowledge the source code of the remote-ssh extension isn't available in unmangled form. Should be trivial to do for the VS code team, though.

[oschulz]

Thanks @RobertRosca , I'll definitely try that, such a wrapper should be possible. Will post if it works.

I played a bit with this, but it's tricky because Singularity (and similar scientific container runtimes like Shifter and Charliecloud) are very different from docker - they don't have a background service, they just run the container directly, there's only one process. So things like docker ps -q don't really exist. It's much closer to a glorified chroot, instead.

[RobertRosca]

@oschulz Hmm good point, there aren't analogs for all of the commands you can do in docker in singularity, but most things are possible, singularity actually has decent OCI Runtime Support built in already.

In Singularity you can run containers as a background service, this is called a container instance instance, you can also run containers as OCI compliant instances which gives you JSON-format outputs for the state of the container which have the pattern OCI expects.

So if you start a container with singularity instance start ..., it runs as a background process, and you can then do singularity instance list to get a list of running instances which is pretty similar to docker ps -q.

Played around a bit with the vscode remote docker extension and the calls it makes are usually like:

[254405 ms] Start: Run: docker context show
[254423 ms] Start: Run: docker context show
[254442 ms] Start: Run: docker context inspect --format {{.Name}}
[254464 ms] Start: Run: docker context inspect --format {{.Name}}
[254496 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[254531 ms] 1.41
[254532 ms] Start: Run: docker -v
[254561 ms] Start: Run: docker events --format {{json .}} --filter event=create --filter event=destroy --filter event=die --filter event=health-status --filter event=kill --filter event=pause --filter event=rename --filter event=resize --filter event=restart --filter event=start --filter event=stop --filter event=unpause --filter event=update --filter event=exec_create
[254568 ms] Start: Run: docker ps -q -a
[254616 ms] Start: Run: docker inspect --type container  # Ran on all containers

That's just done as the setup, now when you select a container to attach to it runs:

[299720 ms] Start: Run: docker inspect --type container aba69f4...
[299760 ms] Start: Run: docker start #  Ran on the container you selected to attach to
[301149 ms] Start: Run: docker inspect --type container aba69f4...

On the 'local' client, and then the following runs on the 'remote' client:

[3 ms] Remote-Containers 0.161.0 in VS Code 1.54.0-insider (a699ffaee62010c4634d301da2bbdb7646b8d1da).
[9 ms] Start: Run: docker context show
[53 ms] Start: Run: docker context inspect --format {{.Name}}
[84 ms] Start: Run: docker context show
[87 ms] Start: Resolving Remote
[88 ms] Setting up container: /vscode_test_container
[89 ms] Start: Run: docker inspect --type container /vscode_test_container
[119 ms] Start: Run: docker context inspect --format {{.Name}}
[133 ms] Start: Check Docker is running
[133 ms] Start: Run: docker version --format {{.Server.APIVersion}}
[170 ms] Server API version: 1.41
[171 ms] Start: Inspecting container
[171 ms] Start: Run: docker inspect --type container aba69f4...
[198 ms] Start: Run: docker exec -i -u root -e VSCODE_REMOTE_CONTAINERS_SESSION=8033a654-1097-4d65-9b42-b6472cfb27df1613987151778 aba69f4... /bin/sh
[201 ms] Start: Run in container: uname -m
[231 ms] Start: Run in container: cat /etc/passwd

You can place various additional flags to be passed to this in a runArgs variable, either with docker-compose or devcontainer files, which would let you set some additional flags for Singularity (e.g. containall).

I've made a (empty) repo here: https://github.com/RobertRosca/singularity-vscode-devcontainer

And an initial dev issue collecting some info: https://github.com/RobertRosca/singularity-vscode-devcontainer/issues/2

[oschulz]

In Singularity you can run containers as a background service

Hm, good point, have to look into that (and check if it requires permissions that plain users on scientific compute clusters might not get, they tend to be restrictive).

[RobertRosca]

Hm, good point, have to look into that (and check if it requires permissions that plain users on scientific compute clusters might not get, they tend to be restrictive).

Should be fine, I've only glanced through the code but as far as I understand it instances work in a pretty simple way. When you start an instance it starts a container as a background process and then (by default) writes the PID and some other info to the users home directory (e.g. ~/.singularity/instances/sing/$HOSTNAME/$USERNAME) in JSON files.

After that all the commands that interact with instances (e.g. list, stop, log, run, attach, etc...) look through that directory to find the PID/other relevant info, and use that PID to communicate with the background container. As long as the user can start singularity containers and the instance JSON files can be written then it should work fine. I currently use singularity instances on the cluster at work and haven't had any issues with them.

[oschulz]

Great, thanks for the details!

[PavelSosin-320]

Hi Robert, It is very interesting! I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment. It requires extra hardware, extra software, extra network equipment, and extra configuration per user without any obvious benefits. It is possible and very easy to run any dockerless OCI runtime locally using natively on the Linux Desktop or on the Windows using WSL. The total memory footprint doesn't exceed 0.5 Gb plus 2Gb taken by WSL service in the case of Buildah.OCI runtimes and OCI image repositories are not Docker's monopoly and can be found today almost in every corner of Global Cloud infrastructure free of charge in many cases. Could you share links to Singularity documentation, Installation guide, and Hardware&Software requirements?

On Mon, Feb 22, 2021 at 4:53 PM Oliver Schulz notifications@github.com wrote:

Great, thanks for the details!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/3066#issuecomment-783429769, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM7YVKFUU5ENCFLWAIYMZJLTAJVVBANCNFSM4NNB5SOQ .

[oschulz]

Should be fine, I've only glanced through the code but as far as I understand it instances work in a pretty simple way.

This might indeed work nicely with singularity. The problem is that other relevant container runtimes like Shifter and Charliecloud do (AFAIK) not support instances.

In the end, a simple option in VS-code that allows for an arbitrary prefix commands before the vscode-server starts (so like it was put before "$ROOT/node [...] in the server start script) would probably be the most flexible solution.

[mathiaswagner]

I used a separate ssh-key and in the remote host's authorized_keys I prefix that special public key key with

command="singularity run image.sif $SSH_ORIGINAL_COMMAND"

This seems to work.

On my local host I have in my .ssh/config

Host vscoderemote
HostName remote
IdentityFile /path/to/ssh-key-for-code

[oschulz]

Neat!

[DraTeots]

@PavelSosin-320 I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment

Well... You don't understand =) Since singularity is in play, it is highly probably a scientific environment, where software part is usually understaffed and software consists of crappy monster packages like CERN ROOT. Some software from the stack is new and demanding (c++17, newest compilers, look at ACTS tracking), some other software is 50 years old, all developers are passed away and nobody touches it as it works but requires old versions of Fortran. End everything is wired together with lots of environment variables, hacks and patches. Building such stack from scratch make take days of fails and gotchas. Images are 7-8GB stuffed of scientific dependencies where "everything works, don't touch". Understaffed maintainers cannot provide sane stably working installation solutions other than containers. Please, try to build something like sPHENIX software stack and then come back to this issue as new person who now understands all.

Docker container support in VSCode is superb and it is pleasure to develop in the container. But in many scientific communities docker is given up in favor of Singularity. Such plugin would greatly improve situation for many scientists.

[oschulz]

I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment [...]It requires extra hardware

It's not so much about plain software development for generic applications (write on your local system, test on your local system, deploy to the cloud or so).

As it turns out, VS code is an great tool for scientific coding and analysis. So code is written, but it's constantly changed, plots are made, data is analysed, code is adapted, and to on. That often requires systems or compute clusters with capabilities far beyond a laptop or desktop machine, and indeed sometimes special hardware like state of the art GPUs, etc.

[oschulz]

Ah, I forgot - also, in typical scientific computing environments (central home directories, no root privileges for users on desktops or severs) Docker can't really be used safely, so Singularity & friends (locally and remote) have become very common.

[PavelSosin-320]

Singularity, Docker, Podman, Buildah, Kubernetes & friends share the same Image standard - OCI. The Image built using one of the listed tools can be run in any other tool almost without effort. The OCI runtime can run on any scale, from Podman Linux CLI to Kubernetes clusters. The client-Server architecture is the personal developer choice, there is no herd race here today. Every developer can make the choice to use everything on a single laptop or use a Cloud-based development pipeline. Base docker images for NVidia GPU are available from NVidia, they can be used on almost any hardware with NVidia GPU, from the laptop to Cloud infrastructure like AWS, GCP. So Creation of the Docker image is a one-time investment that makes software portable forever. My choice today is Podman at home and RedHat dev pipeline for big projects with complex integration. Tlalks about Docker safety is bullshit today, Banks and Government defense bodies use Kubernetes because it offers better security frameworks.

[PavelSosin-320]

Oliver, the Podman development is lead by the person with the nickname "SELinux man". So, you can use it safely or report your doubts directly.

On Sun, May 23, 2021 at 10:02 AM Oliver Schulz @.***> wrote:

Ah, I forgot - also, in typical scientific computing environments (central home directories, no root privileges for users on desktops or severs) Docker can't really be used safely, so Singularity & friends (locally and remote) have become very common.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/vscode-remote-release/issues/3066#issuecomment-846515126, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM7YVKDOVC3JOY35V6WPNHTTPCR7PANCNFSM4NNB5SOQ .

[DraTeots]

The OCI runtime can run on any scale

Can run. But also can not to run. Just pure day to day experience here. No theory-crafting. You are very good at trying to describe to us, that we don't have problems. But we do have them. And would like to have a solution. That is about it.

[PavelSosin-320]

This requirement consists of two independent parts:

1. Convert Microsoft base container OCI image into Singularity format. This process is described in the Singularity documentation and has to be addressed to the Singularity dev team if you have any problems.
2. Ability to run Singularity CLI / Mesos client CLI instead of docker CLI. VSCode implements the practical approach allowing aliasing of CLI name, like for Podman. You can try it and estimate whether it is feasible. Unfortunately, there is no standard for CLI syntax but if in the case of similarity of the Docker and Singularity commands it may work.
3. In the Singularity documentation I found the firm promise to support OCI. If something doesn't work follow p 1.
4. Devcontainer plugin converts devcontainer.json file into Dockerfile syntax, then runs build command followed by docker -it run or Docker-compose up. If image definition languages are not compatible this is a Request for a new plugin. Build process extensibility is not supported by VSCode for any image builders: buildkit, buildah.

[PavelSosin-320]

I don't understand what is the goal of the herd race toward already obsolete Client-Server architecture in the development environment [...]It requires extra hardware

It's not so much about plain software development for generic applications (write on your local system, test on your local system, deploy to the cloud or so).

As it turns out, VS code is an great tool for scientific coding and analysis. So code is written, but it's constantly changed, plots are made, data is analysed, code is adapted, and to on. That often requires systems or compute clusters with capabilities far beyond a laptop or desktop machine, and indeed sometimes special hardware like state of the art GPUs, etc.

Those who don't like Client-Server architecture the way is out of VSCode because VSCode is essentially Client (Electron) - Server NodeJS. For pure Cloud application developers who like serverless freedom, RedHat offers Podman that just allowed me to create rootless Pontainer-based service, generate systemd unit out of the service and start it without any client just upon my user logon via systems-user.

[oschulz]

@PavelSosin-320 Oliver, the Podman development is lead by the person with the nickname "SELinux man". So, you can use it safely or report your doubts directly.

I think this is a misunderstanding - of course Docker, Kubernetes & friends are run securely at very large scales today. But they are deployed for cloud-style workflows that are mainly about running services. Also, deployment is typically controlled by people with administrative privileges. This is very different from the scientific computing environment were a large number of users bring their own containers, run them in interactive mode or as part of jobs on batch systems. More and more supercomputer sites now offer containers to users, with various runtimes (mainly Singuarity, Shifter and Charliecloud). These have to work together with both GPUs and MPI, there is a large number of users that are both developers and end users at the same time, etc. To my knowledge, no scientific computing site currently offers Docker-style runtimes to end users for batch jobs and interactive work - it's just not what Docker & friends are built for.

[PavelSosin-320]

@oschulz GCP offered both NVidia and HANA Kubernetes nodes for both business and scientific ( calculative genetic ) purposes. I know it for sure because.. . 3 months of super fast Genetic training were amazing.

[oschulz]

GCP offered both NVidia and HANA Kubernetes

Well, usually we can't choose our computing resources, since they tend to depend on the project and funding-channel(s). :-)

In any case, I don't see why adding support for Singularity and similar container runtimes would be a bad thing.

[PavelSosin-320]

@oschulz maybe you chose the wrong address? VSCode is not an open-source project. This is an Open source product with a certain backlog, priorities, and budget. Full support of Singularity containers may cost a fortune due to their complexity and needs a long time. Microsoft is not responsible for Docker and selected Docker Desktop for Win only because there were no available alternatives. Docker engine is the server running on Linux, dedicated to mass request processing. Docker Desktop for Win is Docker CE server running on Alpine VM If you need GPU you can use one of the available Linux desktops distros with better GPU support than Windows if it not used for games. Many CAD systems can run on Linux desktops. You can use VSCode-remote and share Linux servers between several developers without compromising security. There is another development environment GitHub workspaces. It is fully cloud-based and doesn't need additional hardware and IT. Maybe, it better suits for research projects.

[oschulz]

VSCode is not an open-source project. This is an Open source product with a certain backlog, priorities, and budget.

@PavelSosin-320, please excuse my ignorance, are you part of the Microsoft VSCode team?

Full support of Singularity containers may cost a fortune due to their complexity and needs a long time.

It might actually be quite easy, since runtimes like Singularity, Shifter and Charliecloud are not service-based, but "simple" commands that do a glorified chroot. A way to implement this could be adding a VS-code remote-SSH option like remote.SSH.runwith that would allow the user to specify a wrapper script for the vscode server. So "remote.SSH.runwith": "run_in_my_container.sh" would result in an equivalent of run_in_my_container.sh $HOME/.vscode-server/bin/.../server.sh. It would need to a setting that can differ for different target-systems, though.

[tbugfinder]

VSCode is not an open-source project. This is an Open source product with a certain backlog, priorities, and budget.

@PavelSosin-320, please excuse my ignorance, are you part of the Microsoft VSCode team?

Full support of Singularity containers may cost a fortune due to their complexity and needs a long time.

It might actually be quite easy, since runtimes like Singularity, Shifter and Charliecloud are not service-based, but "simple" commands that do a glorified chroot. A way to implement this could be adding a VS-code remote-SSH option like remote.SSH.runwith that would allow the user to specify a wrapper script for the vscode server. So "remote.SSH.runwith": "run_in_my_container.sh" would result in an equivalent of run_in_my_container.sh $HOME/.vscode-server/bin/.../server.sh. It would need to a setting that can differ for different target-systems, though.

I'd like to second that. It might be easier to add a configuration of what's being executed on a specific target system defaulting to some docker environment. I don't consider Github Workspaces as an option.

[PavelSosin-320]

@oschulz No, I'm not a part of MS stuff. But in the course of my long carrier, I participated in the 6 Project focused on Development environment development. This is my expertise. So, I can estimate the complexity and efforts required to fulfill such feature requests. Only the Differences between Docker and Singularity shows how these 2 container runtime systems are different including API, image definition language, etc. I believe that it is a better tool for a certain area ( like Hana studio plugin is better for big data processing applications development ). I see how much it can cost.

[oschulz]

@PavelSosin-320: No, I'm not a part of MS stuff.

My impression so far was that the MS VS-Code team doesn't mind (and may in fact appreciate) constructive discussions about feature ideas (like Singularity support) here on GitHub. The decision on what features to implement and what resources to allocate for that certainly lies with them, of course, not with us.

[PavelSosin-320]

Just an update, I can start the vscode-server inside the singularity container. It starts without any issue. The problem now is how to forward ssh or config the ssh extension locally to use this server.

If you can start vscode-server inside singularity container you can start 2 containers as a backend: devcontainer with tools plus vscode server. that share the workspace. This architecture was proposed by the VSCode project leader Erich Gamma and not is new. The side-car architecture is implemented over the Cloud as Eclipse/Che (Kubernetes pods), SAP Web IDE Cloud Foundry app+sidecar). The problem is always how to share workspace. But if in Singularity everything is shared then it is not a problem. Since Singularity use CNI (bravo!) IPC is not a problem too.

[oschulz]

Singularity & friends (Shifter, Charliecloud, etc.) are typically configured to bind-mount the user's home and all relevant network storage into the container instance, so basically just "/usr" and the like get swapped out. In the default configuration, processes running in Singularity even share PID space with processes outside, there's also no network isolation, etc. It's really mostly a glorified chroot to bring in a well-defined software stack but have the same environment otherwise. Singularity can do virtual networking now, I think, but it's typically not used for scientific computing, resp. batch jobs. I'm not sure the VS-Code devcontainer approach is compatible with that - but we don't really need it. If devcontainer.json would support Singularity & Co., that might be a convenient way to tell VS-Code which image to use though. But any way of telling VS-Code to run it's server component via a user-defined startup script would make it compatible with all these container runtimes.

[PavelSosin-320]

@oschulz Out of this thread but FYI. If you need only IDE "frame" for code editing you can try to run Theia. If you run you can simply drop one line command (for Podman): podman run -it -p 3000:3000 -v "$(pwd):/home/project:cached" theiaide/theia:next . In any other OCI compatible tool it will be something very similar. If you will see this as helpful, you can connect me vi pavel.sosin@ .... No installation is required, just Chrome.

[winni2k]

I got @oschulz 's solution (https://github.com/microsoft/vscode-remote-release/issues/3066#issuecomment-841305890) to work as well, but I had to replace singularity run with singularity shell because my Rocker R-base container runs R by default. Here is what worked for me:

In my remote host's .ssh/authorized_keys file I used this prefix for the special public key:

command="singularity shell image.sif $SSH_ORIGINAL_COMMAND"

[oschulz]

I also changed my approach in the mean time. My current workaround is to use a special singularity-ssh script

#!/bin/sh

# Get last command line argument, should be hostname/alias
for trghost; do true; done

# Parse host-aliases of form "venvname~hostname"
imagename=`echo "${trghost}" | sed 's/^\(\(.*\)~\)\?.*$/\2/'`

# Note: VS Code will override "-t" option with "-T".

if [ "${imagename}" = image1 ]; then
    exec ssh -t "$@" singularity shell /path/to/image1.sif
elif [ "${imagename}" = image2 ]; then
    exec ssh -t "$@" singularity shell /path/to/image1.sif
else
    exec ssh "$@"
fi

and to put entries like this in my `$HOME/.ssh/config"

Host somehost image1~somehost image2~somehost
  HostName somehost.some.domain
  # ...

So singularity-ssh hostname will do a normal SSH connection to the host, but singularity-ssh image1~hostname will result in a shell running in a singularity instance on the remote system.

Now add

"remote.SSH.path": "/path/to/singularity-ssh"

to your VS-Code settings (must be an absolute path or VS-Code will silently ignore it!)

With that set up, you can just select somehost, image1~somehost or image2~somehost in "Remote-SSH: Connect to host" in VS-Code.

No manipulation of $HOME/.vscode-server/bin/.../server.sh or any other action on the remote system necessary anymore. :-)

I still haven't found a way to run several VS-code remote SSH sessions on the same host with different container images simultaneously, though.

[DebajyotiS]

@oschulz I have been banging heads against a wall trying to get VS Code recognize modules installed inside a singularity image and debug, on a remote machine for ages. Your solution above looks like it would work for me. Could your script be modified in a way so that I can point to which node I want to ssh to? I believe this, as is, would ssh to the login node (?), which might not be ideal for debugging large projects. The cluster I have access to, has dedicated debug nodes, and so I was wondering if there's a way to point to that.

[oschulz]

The cluster I have access to, has dedicated debug nodes, and so I was wondering if there's a way to point to that.

Yes, that should be possible using the ssh ProxyCommand setting. Put something like this in your .ssh/config:

Host compute-center-login-node
  HostName login-node.some.compute.center
  User YOUR_USER_NAME

Host debug-node-1 myimage~debug-node-1
  HostName debug-node-1.some.compute.center
  ProxyCommand compute-center-login-node mpp -W %h:22
  User YOUR_USER_NAME

An ssh debug-node-1 will now internally connect to compute-center-login-node first, and then to debug-node-1 from there. It'll look like a direct SSH connection from your side (you may have to enter your password twice, though).

So now you should be able to Remote-SSH-connect directly to debug-node-1 from VS-Code, respective to myimage~debug-node-1 to get a VS-Code server running in singularity image myimage on debug-node-1 (using the singularity-ssh solution described above).

[DebajyotiS]

Ah, neat!

If I have multiple projects, each with their own singularity images, and if I want to open one of them at any given time for debugging, would this translate to:

a) adding all the image paths as you show in singularity-ssh script. b) in the .ssh config file creating individual aliases so that I can identify them when I select them from the 'connect to host' menu in vs code?

[oschulz]

If I have multiple projects, each with their own singularity images [...]

Yes, I do use it like that. Just haven't figured out how to use several images on one remote host at the same time, since VS-Code re-uses a running server instance.

[DebajyotiS]

So I think I am doing something wrong. I create a singularity-ssh.sh file in my D:\ drive, like so:

#!/bin/sh

# Get last command line argument, should be hostname/alias
for trghost; do true; done

# Parse host-aliases of form "venvname~hostname"
imagename=`echo "${trghost}" | sed 's/^\(\(.*\)~\)\?.*$/\2/'`

# Note: VS Code will override "-t" option with "-T".

if [ "${imagename}" = xrim ]; then
    exec ssh -t "$@" singularity shell /path/to/sif
else
    exec ssh "$@"
fi

And in my config file:

Host baobab xrim~baobab
    HostName hostname@server
    User userid

and the remote.ssh.path:

D:\singularity-ssh.sh

Thereafter, when I try to connect to host named xrim - it complains that the spawn is unknown.

[oschulz]

I create a singularity-ssh.sh file in my D:\ drive

Are you on Windows? If so, my "singularity-ssh.sh" Linux shell script will not work for you, you'll need a Windows-shell equivalent.

[DebajyotiS]

I tried modifying the script here:

 cat singularity-ssh.sh
#!/bin/sh
# Get last command line argument, should be hostname/alias
# Parse host-aliases of form "venvname~hostname"
imagename=$(echo "$1" | sed 's/^\(\(.*\)~\)\?.*$/\2/')
echo "$imagename"

if [ "$imagename"==xrim ]
then
    echo "image is xrim"
    ssh -t user@host.server "cd go/to/path/ && singularity shell /path/to/sif"
else
echo "image is not xrim"

which works fine from the shell when I do bash singularity-ssh.sh xrim~baobab, it sshs in, cds to the directory and fires up the singularity iamge.

However from vs-code I still get the spawn unkonwn error.

Any clue what's going on? I reinstalled the remote development and remote ssh extensions but to no avail.

EDIT: Renamed my script to singualrity-ssh and made it executable (and updated the path accordingly) - now it is able to ssh, but doesn't fire up the singularity image or cd into the right directory. (works fine from a terminal though) How is VS Code actually using this singularity-ssh script?

[oschulz]

now it is able to ssh, but doesn't fire up the singularity image

Are you sure the path in VS-Code is correct? The problem is, if VS-Code doesn't like the custom SSH program, it seems to silently ignore it and use the regular one.

Also, are you sure you killed all tasks related to the VS-Code server on the remote system? If there's still an old server instance running, it will be reused.

What kind of system are you using the the client (VS-Code user interface) side?