VSCode 1.82 and Remote 0.106.1 (or earlier?) compatibility

[Dridus]

It looks like today's VSCode 1.82 release changes the location that the servers are deployed and thus breaks this.

Previously, VSCode deployed servers to ~/.vscode-server/bin but now it seems that perhaps the remote tunnel support is on by default or mandatory or just something similar is in use as it now deploys them to ~/.vscode-server/cli:

[ross@BEAST:~]$ find .vscode-server -type f -o \( \( -name logs -o -name data -o -name out -o -name node_modules -o -name extensions \) -prune \)
.vscode-server/code-8b617bd08fd9e3fc94d14adb8d358b56e3f72314
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/extensions
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/node_modules
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/out
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/package.json
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/product.json
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/LICENSE
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/bin/remote-cli/code
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/bin/code-server
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/bin/helpers/browser.sh
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/node
.vscode-server/.cli.8b617bd08fd9e3fc94d14adb8d358b56e3f72314.log

[ross@BEAST:~]$ file .vscode-server/code-8b617bd08fd9e3fc94d14adb8d358b56e3f72314
.vscode-server/code-8b617bd08fd9e3fc94d14adb8d358b56e3f72314: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=f3a1f0cf7d19de3990a25de103f3122353c43245, stripped

[ross@BEAST:~]$ ldd .vscode-server/code-8b617bd08fd9e3fc94d14adb8d358b56e3f72314
        statically linked

[ross@BEAST:~]$ file .vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/node
.vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/node: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=882ca1ed9ae43f8b1272528f36bb6e1b19df9829, with debug_info, not stripped

[ross@BEAST:~]$ ldd .vscode-server/cli/servers/Stable-8b617bd08fd9e3fc94d14adb8d358b56e3f72314.staging/server/node
        linux-vdso.so.1 (0x00007ffc889aa000)
        libdl.so.2 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/libdl.so.2 (0x00007f9753687000)
        libstdc++.so.6 => not found
        libm.so.6 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/libm.so.6 (0x00007f97535ae000)
        libgcc_s.so.1 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/libgcc_s.so.1 (0x00007f9753594000)
        libpthread.so.0 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/libpthread.so.0 (0x00007f975358f000)
        libc.so.6 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib/libc.so.6 (0x00007f975338f000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/scd5n7xsn0hh0lvhhnycr9gx0h8xfzsl-glibc-2.34-210/lib64/ld-linux-x86-64.so.2 (0x00007f975368e000)

Following on the name cli I thought maybe the vscode-cli branch from #63 might help, but that targets .vscode-cli and I don't think works with this new deeper directory structure, and is also broken. I fixed it up in https://github.com/VitalBio/nixos-vscode-server/tree/vscode-cli before realizing that it wasn't the same thing.

We need this to work so I'm taking a crack at it but if anybody has a fix tucked away I'd gladly accept ;-)

cc @asariley @dfithian

[msteen]

Thanks for letting me know, despite being the author of this project, I hardly use it myself. I just updated VS Code to 1.82 and reloaded Remote to let it use 0.106.1, yet it still works for me. Even removed .vscode-server just in case caching was the cause, but it is still working. Tried installing Remote - Tunnels, but having it doesn't affect it for me. I am on Ubuntu connecting with a NixOS server. Do you have any additional information on your setup that might explain why it broke for you with the latest update while it is still working for me?

Regardless, I think it is high time to replace the current approach for something more general, more general than being specific for VS Code Server or CLI. I think I should just make the watcher recursive for new files, and if they are executable, patch them, and make it possible to configure it so you can specify what libraries to make available for patching, and also opt to replace binaries like node, rather than just patching them. And then use that project to implement this project for ease of use / backwards compatibility.

[leolavaur]

I can confirm it is broken for me, with VS Code 1.82.0. I'm on a Mac, connecting to a NixOS server. The service is deployed using home-manager and the last flake revision.

[msteen]

And you use the Remote - SSH; Connect to Host command? I checked on my Windows machine too, it still works there too. Both are using version 0.106.2 though I highly doubt a minor version should be the difference maker. Kinda want it to break on my end too, so I can more easily verify my fixes work. Though in the worst case I just ask any of you to verify whether my fixes work when I am done and I have tested it still works for the older directory structure (should be general enough to work then too for the new structure).

[pjjw]

Yeah, broken for me, too. I just downgraded vscode to 1.81 and it worked. totally different folder structure and also it looks like it's actively checking if the binary has been replaced, wasn't able to manual-patch it either, but I only tried once, so could have been getting it wrong.

[leolavaur]

And you use the Remote - SSH; Connect to Host command?

Yep, but with the extension version at 0.106.1 in my case.

[msteen]

If I have the time I will try if I can reproduce it in a clean setup with some VMs, but since I cannot reproduce the new situation at the moment, could others that experienced this problem verify whether PR #68 fixes things for them?

I am working on a nix-ld alternative sorts, based on inotifywait watchers that patch newly installed binaries. As I am unhappy with having to change the patch logic every time Microsoft chooses to change something that impacts it.

[Dridus]

PR #68 doesn't currently build for me:

building '/nix/store/h5qn2cnf9a2dzf3sa5xzkqsi3y2r1agb-auto-fix-vscode-server.drv'...

In /nix/store/fqqrd4kzbs9s046mv9p6asaiic5d0dqq-auto-fix-vscode-server/bin/auto-fix-vscode-server line 17:
  if [[ $bins_dir == "~/.vscode-server/cli/servers" ]]; then
                      ^--------------------------^ SC2088 (warning): Tilde does not expand in quotes. Use $HOME.

For more information:
  https://www.shellcheck.net/wiki/SC2088 -- Tilde does not expand in quotes. ...
error: builder for '/nix/store/h5qn2cnf9a2dzf3sa5xzkqsi3y2r1agb-auto-fix-vscode-server.drv' failed with exit code 1

I'll poke around getting that fixed but figured I'd give an update

[Dridus]

Okay, I tried more and ran into more issues, and eventually also got unhappy with having to change the patch logic. Thanks @msteen for the tip about nix-ld, as I used that with some VSCode side configuration instead of this repo and it works for me though it's more limited.

I put this in my server configuration.nix:

{ pkgs, ... }: {
  ...

  programs.nix-ld.enable = true;

  services.openssh.extraConfig = ''
    AcceptEnv NIX_LD NIX_LD_LIBRARY_PATH
  '';

  environment.etc."vscode-ssh-support".source = pkgs.stdenv.mkDerivation {
    name = "vscode-ssh-support";
    phases = ["installPhase"];
    installPhase = ''
      set -e
      mkdir -p $out
      for lib in ${pkgs.glibc}/lib/ld-linux* ${pkgs.stdenv.cc.cc.lib}/lib/*; do
        ln -sf $lib $out/$(basename $lib)
      done
    '';
  };

  ...
}

and then setting the "Remote - SSH: Config File" to a fresh custom thing that sets the environment:

Host place
  User ross
  SetEnv "NIX_LD=/etc/vscode-ssh-support/ld-linux-x86-64.so.2" "NIX_LD_LIBRARY_PATH=/etc/vscode-ssh-support"

admittedly it's a lot more of a random grab-bag of configuration than something nice, but it seems to work

[msteen]

Thanks for checking! For #68 to work you will have to change your installPath and replace ~ with $HOME. Something that would need fixing if I am to merge it, but that change should likely make it build for you.

Also thanks for sharing an example of how things could be made to work with nix-ld. This gives people at least an alternative until we get things working again.

[Dridus]

yes, I changed the ~ out and that helped. I also made a switch for whether the old-style bin or new-style cli was the intent as before the directory detection would assume old style and mkdir that if vscode had not already created the new structure.

after doing those two things I ran into what looked like sequencing issues where it tried to move a nonexistent file during staging? at that point I made the call to switch to the approach of making vscode run things in an environment that would let it run unpatched binaries rather than sorting out the precise sequence of events to correct the inotify+patch approach

[MForster]

Thanks for the tip with nix-ld. Indeed, all that I needed in my case was

  programs.nix-ld.enable = true;

[Ten0]

Well here we are, seems this is now released as part of non-insiders version and it's now broken for ~everybody. Half of my team can't work on their dedicated remote computer 😶

Sample 1: OS: Mac VSCode version: 1.87.0 Remote SSH version: 0.109.0

This branch works for me: https://github.com/Ten0/nixos-vscode-server/commits/master/

[bebert64]

Well here we are, seems this is now released as part of non-insiders version and it's now broken for ~everybody. Half of my team can't work on their dedicated remote computer 😶

Sample 1: OS: Mac VSCode version: 1.87.0 Remote SSH version: 0.109.0

This branch works for me: https://github.com/Ten0/nixos-vscode-server/commits/master/

Hello everyone. As part of the mentioned team, I'd like to share my (non-working) setup as well:

Sample 2: OS (local): 6.7.9-arch1-1 VSCode: 1.87.1 Remote - SSH: 0.109.0

[msteen]

@Ten0 I haven't used this project myself in over a year, despite doing some work on it. I asked for other maintainers on Discourse, but no one replied. You seem to have a vested interest and seem able to patch things a necessary, would you or anybody on your team be willing to become a maintainer?

You mentioned your branch works for you, if so, could you turn it into a PR?

[Ten0]

You mentioned your branch works for you, if so, could you turn it into a PR?

That version was essentially based on #68, but I quickly started running into issues - it looks like that PR went the wrong way about handling the directory renames that vscode performs during installation.

You seem to have a vested interest and seem able to patch things a necessary, would you or anybody on your team be willing to become a maintainer?

In that spirit and given the state of #68 I took some time to rewrite a PR that works from ~scratch, also changing the directory structure so that it hopefully makes it easier to support future VSCode directory structure changes. That is #78.

I guess me being a maintainer couldn't hurt if you're not dedicating time to this anymore 🙂 I may not be able to always prioritize reviewing things here, but at least that would increase the bus factor. 👍️