search

microsoft / vscode-remote-release

Visual Studio Code Remote Development: Open any folder in WSL, in a Docker container, or on a remote machine using SSH and take advantage of VS Code's full feature set.
https://aka.ms/vscode-remote
Other
3.61k stars 274 forks source link

[Remote-SSH Bug]: spawn EINVAL error when running ssh through batch file on Windows #10061

Closed xangma closed 4 weeks ago

xangma commented 1 month ago

Is there an existing issue for this bug?

Required Troubleshooting Steps

Connect Locally

It connects successfully

->

I'm using vscode-insiders v1.92 with remote-ssh v0.113.2024070816 (pre-release).

After installing node v22.4.1 and npm 10.8.1 on a Windows 11 machine, when I use the Remote SSH extension to start ssh from a batch file, it now errors with:

[12:14:16.907] Install and start server if needed
[12:14:16.977] Checking ssh with "C:\Users\username\ssh.bat -V"
[12:14:16.978] Finding installed ssh failed: spawn EINVAL

This is issue the same as this issue, where they explain the reason:

see https://github.com/nodejs/node/issues/52554 and https://nodejs.org/en/blog/vulnerability/april-2024-security-releases-2

workaround is to add { shell: true }

I can confirm that this works when changing it in extension.js here:

        async function R(e, t) {
          return new Promise((n, o) => {
            t.debug(`Checking ssh with "${e} -V"`);
            const i = [],
              s = [],
              a = r.spawn(e, ["-V"], { shell : true });
            a.stdout.on("data", (e) => {
              i.push(e), t.debug("stdout> " + e.toString());
            }),

and in localServer.js here:

      static spawnSsh() {
        h(
          `Running ssh connection command: ${p.sshCommand} ${p.sshArgs.join(
            " "
          )}`
        );
        const e = t.spawn(p.sshCommand, p.sshArgs, {
          stdio: ["inherit", "pipe", "pipe"],
          windowsHide: !0,
          shell : true , 
        });
        let r = !1;

And now the connection works again. I'm sure there's probably security considerations etc., but I think these files aren't open sourced anyway so this is where my investigation ends sadly.

EDIT: I'm pretty sure there's more places that shell : true are needed, for full coverage, but this is the minimum I needed to get connecting again so I went with it.

Expected Behavior

It connects fine.

Steps To Reproduce

Set up the Windows environment the same and try to connect using a batch script that runs ssh.

Remote-SSH Log

Remote-SSH Log

``` [12:40:08.934] Log Level: 1 [12:40:08.946] VS Code version: 1.92.0-insider [12:40:08.946] Remote-SSH version: remote-ssh@0.113.2024070816 [12:40:08.946] win32 x64 [12:40:08.964] SSH Resolver called for "ssh-remote+len", attempt 1 [12:40:08.964] "remote.SSH.useLocalServer": true [12:40:08.964] "remote.SSH.useExecServer": true [12:40:08.964] "remote.SSH.path": C:\Users\username\ssh.bat [12:40:08.964] "remote.SSH.configFile": undefined [12:40:08.964] "remote.SSH.useFlock": true [12:40:08.964] "remote.SSH.lockfilesInTmp": true [12:40:08.965] "remote.SSH.localServerDownload": off [12:40:08.965] "remote.SSH.remoteServerListenOnSocket": false [12:40:08.965] "remote.SSH.showLoginTerminal": true [12:40:08.965] "remote.SSH.defaultExtensions": [] [12:40:08.965] "remote.SSH.loglevel": 1 [12:40:08.965] "remote.SSH.maxReconnectionAttempts": 0 [12:40:08.965] "remote.SSH.enableDynamicForwarding": true [12:40:08.965] "remote.SSH.enableRemoteCommand": true [12:40:08.965] "remote.SSH.serverPickPortsFromRange": {} [12:40:08.965] "remote.SSH.serverInstallPath": {} [12:40:09.000] SSH Resolver called for host: len [12:40:09.000] Setting up SSH remote "len" [12:40:09.003] Acquiring local install lock: C:\Users\username\AppData\Local\Temp\vscode-remote-ssh-366adb0c-install.lock [12:40:09.004] Looking for existing server data file at c:\Users\username\AppData\Roaming\Code - Insiders\User\globalStorage\ms-vscode-remote.remote-ssh\vscode-ssh-host-366adb0c-a35380d6f0ff65f7c35345baafcf1c3f8a378d38-0.113.2024070816-es\data.json [12:40:09.004] Using commit id "a35380d6f0ff65f7c35345baafcf1c3f8a378d38" and quality "insider" for server [12:40:09.006] Install and start server if needed [12:40:09.039] Checking ssh with "C:\Users\username\ssh.bat -V" [12:40:09.040] Finding installed ssh failed: spawn EINVAL [12:40:09.042] Resolver error: Error: spawn EINVAL at ChildProcess.spawn (node:internal/child_process:421:11) at Object.spawn (node:child_process:799:9) at c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37250:21 at new Promise () at R (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37246:18) at k (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37243:25) at O (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37364:23) at T (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37231:26) at t.getSshCommandPath (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37353:26) at P.create (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:36083:59) at async e (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:35734:17) at async c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:36877:25 at async t.withShowDetailsEvent (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:37067:22) at async c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:35611:20 at async T (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:35503:17) at async t.resolveWithLocalServer (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:35589:18) at async k (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:36751:21) at async t.resolve (c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:36938:57) at async c:\Users\username\.vscode-insiders\extensions\ms-vscode-remote.remote-ssh-0.113.2024070816\out\extension.js:51368:23 [12:40:09.076] TELEMETRY: {"eventName":"resolver","properties":{"osReleaseId":"","arch":"","askedPw":"0","askedPassphrase":"0","asked2fa":"0","askedHostKey":"0","remoteInConfigFile":"1","gotUnrecognizedPrompt":"0","dynamicForwarding":"1","localServer":"1","didLocalDownload":"0","installUnpackCode":"0","outcome":"failure","reason":"Unknown","exitCodeLabel":""},"measures":{"resolveAttempts":1,"isExecServer":0,"timing.totalResolveTime":77}} [12:40:09.077] ------ ```

Anything else?

No response

josemonsalve2 commented 1 month ago

Facing the same issue.

eickr commented 1 month ago

I'm also receiving this error after updating VS Code to v1.92.0. In the meantime I've partially worked around the issue by removing the "remote.SSH.path" configuration from my settings.json, but my workflow is negatively impacted regardless.

wadebev11 commented 1 month ago

Looks like something released in the new update. I ran into this after updating to v1.92.0. I rolled back to v1.91.1 and it works now

https://code.visualstudio.com/updates/v1_91

samueldyoung29ctr commented 1 month ago

Can confirm that I am seeing the same issue on v1.92 for Windows x64, specifically these lines in the log:

[12:40:09.039] Checking ssh with "<path to my .bat file> -V"
[12:40:09.040] Finding installed ssh failed: spawn EINVAL

and that downgrading to v1.91.1 restores functionality.

blaylockbk commented 1 month ago

Also having the same issue after upgrading to 1.92.0.

alexAtQuanos commented 1 month ago

Same problem here

enszhou commented 1 month ago

same issue after upgrading to 1.92.0.

jakkes commented 1 month ago

Same issue after v1.92.0

thecopeeninja commented 1 month ago

Facing the same issue

Version: 1.92.0 (user setup)
Commit: b1c0a14de1414fcdaa400695b4db1c0799bc3124
Date: 2024-07-31T23:26:45.634Z
Electron: 30.1.2
ElectronBuildId: 9870757
Chromium: 124.0.6367.243
Node.js: 20.14.0
V8: 12.4.254.20-electron.0
OS: Windows_NT x64 10.0.22631
blaylockbk commented 1 month ago

EDIT: I'm pretty sure there's more places that shell : true are needed, for full coverage, but this is the minimum I needed to get connecting again so I went with it.

FWIW: I only needed to change C:\Users\<USER>\.vscode\extensions\ms-vscode-remote.remote-ssh-0.112.0\out\extension.js to make a successful connection. Didn't need to edit localServer.js.

t.debug(`Checking ssh with "${e} -V"`);
            const i = [],
              s = [],
-             a = r.spawn(e, ["-V"]);
+             a = r.spawn(e, ["-V"], { shell: true });
            a.stdout.on("data", (e) => {
              i.push(e), t.debug("stdout> " + e.toString());
            }),
weikinhuang commented 1 month ago

This is also happening to me as well, and adding { shell: true } as @xangma fixes the issue for me

roblourens commented 1 month ago

Sorry for the trouble, this is indeed continuing fallout from the node CVE https://nodejs.org/en/blog/vulnerability/april-2024-security-releases-2 as the OP points out. I'll have a pre-release out soon with a fix.

Just wondering, what is in the .bat file that you're using? I assume you're doing this to customize some arguments for ssh?

weikinhuang commented 1 month ago

@roblourens The ssh.bat is a continuation of this https://github.com/microsoft/vscode-remote-release/issues/937. It would be great if this extension can take into account being able to use the WSL version of ssh (so we can use the ssh config from wsl) natively at some point.

blaylockbk commented 1 month ago

Just wondering, what is in the .bat file that you're using? I assume you're doing this to customize some arguments for ssh?

@roblourens My ssh.bat uses Kerberos plink.exe to make a connection to the remote instead of ssh.exe. See https://github.com/microsoft/vscode-remote-release/issues/250#issuecomment-896215333

roblourens commented 1 month ago

Please install the latest Insiders 1.93 (https://code.visualstudio.com/insiders/) and the latest pre-release v0.114.2024080218 of the extension, it has a fix for this issue. Would appreciate if some of you can confirm that it fixes this for you.

blaylockbk commented 1 month ago

Thanks for the quick fix!

My ssh.bat file looks something like this...

if "%~1" == "-V" GOTO :version

REM If connecting to 'my.remote.host.address' then use plink, otherwise use ssh for all other connections
echo %~4 | find "my.remote.host.address" >nul
if %errorlevel% == 0 ( GOTO :use_plink ) else ( GOTO :use_ssh )

:version
  echo OpenSSH
  GOTO :eof

:use_plink
  /path/to/plink.exe -l <MY_USERNAME> %*
  GOTO :eof

:use_ssh
  ssh.exe %*
  GOTO :eof

The :use_plink section made a successful connection 🎉

The :use_ssh section did not with this error...

[11:27:00.391] Expecting scp to be next to ssh at C:\Users\blaylock\scp.exe
[11:27:00.391] Testing scp with "C:\Users\blaylock\scp.exe"
[11:27:00.393] Got error from scp: spawn C:\Users\blaylock\scp.exe ENOENT
[11:27:00.394] Resolver error: Error: scp installation not found

I couldn't move my ssh.bat file from C:\Users\%USERNAME%\ssh.bat to C:\Windows\System32\OpenSSH, because I don't have admin privileges.

But when I changed the ssh.bat to use Git for Windows' version of ssh.exe, and I moved my ssh.bat file here:

C:\Users\%USERNAME%\AppData\Local\Programs\Git\usr\bin\ssh.bat

then the connection worked. 😄

shiyexuan commented 1 month ago

EDIT: I'm pretty sure there's more places that shell : true are needed, for full coverage, but this is the minimum I needed to get connecting again so I went with it.

FWIW: I only needed to change C:\Users\<USER>\.vscode\extensions\ms-vscode-remote.remote-ssh-0.112.0\out\extension.js to make a successful connection. Didn't need to edit localServer.js.

t.debug(`Checking ssh with "${e} -V"`);
            const i = [],
              s = [],
-             a = r.spawn(e, ["-V"]);
+             a = r.spawn(e, ["-V"], { shell: true });
            a.stdout.on("data", (e) => {
              i.push(e), t.debug("stdout> " + e.toString());
            }),

It works for me!!!!

nkemnitz commented 1 month ago

Please install the latest Insiders 1.93 (https://code.visualstudio.com/insiders/) and the latest pre-release v0.114.2024080218 of the extension, it has a fix for this issue. Would appreciate if some of you can confirm that it fixes this for you.

Thanks, that works as well as the manual edit to extension.js.

My ssh.cmd only contains wsl ssh %* to get ControlMaster support.

ZiyanREN-666 commented 1 month ago

thanks! it works!

xangma commented 1 month ago

Sorry for the trouble, this is indeed continuing fallout from the node CVE https://nodejs.org/en/blog/vulnerability/april-2024-security-releases-2 as the OP points out. I'll have a pre-release out soon with a fix.

Just wondering, what is in the .bat file that you're using? I assume you're doing this to customize some arguments for ssh?

Hello! I'm using the ssh batch file to essentially do a MITM attack to trick vscode into running vscode-server where I want it to run. To do this, I have code to:

I have a repo with the code to do this here. This means we can interactively debug code on our cluster compute nodes, which are hidden behind login nodes, instead of vscode-server starting on the first remote host it lands on. I originally wrote this code because our slurm cluster config was a bit weird, but it seems to work work most configs :-)

fecorreiabr commented 1 month ago

Please install the latest Insiders 1.93 (https://code.visualstudio.com/insiders/) and the latest pre-release v0.114.2024080218 of the extension, it has a fix for this issue. Would appreciate if some of you can confirm that it fixes this for you.

This really fixes the issue, but now I'm facing disconnection issues when using Jupyter Notebook along with vscode-remote, after some minutes of opening the notebook file in the editor:

10:49:48.414 [info] Kernel successfully started
10:49:48.423 [info] Process Execution: #####/venv/bin/python /home/~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2024.7.0-linux-x64/pythonFiles/printJupyterDataDir.py
10:52:46.998 [info] Dispose Kernel '#####.ipynb' associated with '####.ipynb'

I don't know however if this issue is related to the vscode-remote pre-release or the jupyter extension itself.

fecorreiabr commented 1 month ago

Please install the latest Insiders 1.93 (https://code.visualstudio.com/insiders/) and the latest pre-release v0.114.2024080218 of the extension, it has a fix for this issue. Would appreciate if some of you can confirm that it fixes this for you.

This really fixes the issue, but now I'm facing disconnection issues when using Jupyter Notebook along with vscode-remote, after some minutes of opening the notebook file in the editor:

10:49:48.414 [info] Kernel successfully started
10:49:48.423 [info] Process Execution: #####/venv/bin/python /home/~/.vscode-server-insiders/extensions/ms-toolsai.jupyter-2024.7.0-linux-x64/pythonFiles/printJupyterDataDir.py
10:52:46.998 [info] Dispose Kernel '#####.ipynb' associated with '####.ipynb'

I don't know however if this issue is related to the vscode-remote pre-release or the jupyter extension itself.

I've figured out that the "dispose kernel" error is related to Continue extension, as described here.

eickr commented 1 month ago

Just wondering, what is in the .bat file that you're using? I assume you're doing this to customize some arguments for ssh?

One of the primary use cases for me is setting environment variables for the scope, such as SSH env vars that I don't want affecting any other SSH sessions.

roblourens commented 1 month ago

I'll do another stable release with this fix tomorrow morning.

xangma commented 1 month ago

Thanks for the quick fix!

My ssh.bat file looks something like this...

if "%~1" == "-V" GOTO :version

REM If connecting to 'my.remote.host.address' then use plink, otherwise use ssh for all other connections
echo %~4 | find "my.remote.host.address" >nul
if %errorlevel% == 0 ( GOTO :use_plink ) else ( GOTO :use_ssh )

:version
  echo OpenSSH
  GOTO :eof

:use_plink
  /path/to/plink.exe -l <MY_USERNAME> %*
  GOTO :eof

:use_ssh
  ssh.exe %*
  GOTO :eof

The :use_plink section made a successful connection 🎉

The :use_ssh section did not with this error...

[11:27:00.391] Expecting scp to be next to ssh at C:\Users\blaylock\scp.exe
[11:27:00.391] Testing scp with "C:\Users\blaylock\scp.exe"
[11:27:00.393] Got error from scp: spawn C:\Users\blaylock\scp.exe ENOENT
[11:27:00.394] Resolver error: Error: scp installation not found

I couldn't move my ssh.bat file from C:\Users\%USERNAME%\ssh.bat to C:\Windows\System32\OpenSSH, because I don't have admin privileges.

But when I changed the ssh.bat to use Git for Windows' version of ssh.exe, and I moved my ssh.bat file here:

C:\Users\%USERNAME%\AppData\Local\Programs\Git\usr\bin\ssh.bat

then the connection worked. 😄

Ah, if the remote host has Internet access, another workaround for scp not being next to ssh is to set LocalServerDownload to False? The option to only download the server on remote.

joshmcorreia commented 4 weeks ago

I'll do another stable release with this fix tomorrow morning.

I can confirm that this was fixed in 1.92.1 😄

nkemnitz commented 4 weeks ago

Just to confirm: The EINVAL error is supposed to be fixed already with the new stable release 1.92.1 + SSH extension v0.113.1? Does not fix it for me. Neither does v0.113.2024072315 (pre-release). Insiders version 1.93 with SSH extension v0.114.2024080218 worked as expected.

Edit: Ah, it was a new issue, coming from the remote with an outdated wget command that does not support --no-config. Fixed by temporarily enabling "Use Curl and Wget Configuration Files"

@roblourens My first attempt to bypass that and use the "Local Server Download" setting did not work, because VS Code will look for scp.exe in the same path as my ssh.cmd. That does exist, but because the extension is .exe (and not .bat or .cmd), the {shell: true} parameter won't get passed, causing the EINVAL error again.

blaylockbk commented 3 weeks ago

@nkemnitz Edit: Ah, it was a new issue, coming from the remote with an outdated wget command that does not support --no-config. Fixed by temporarily enabling "Use Curl and Wget Configuration Files"

I just ran into this too! Thanks for the suggestion to enable that obscure setting.

I also got around this by installing a recent version of wget in my conda environment rather than relying on the host's older version.

xangma commented 1 week ago

Just to confirm: The EINVAL error is supposed to be fixed already with the new stable release 1.92.1 + SSH extension v0.113.1? Does not fix it for me. Neither does v0.113.2024072315 (pre-release). Insiders version 1.93 with SSH extension v0.114.2024080218 worked as expected.

Edit: Ah, it was a new issue, coming from the remote with an outdated wget command that does not support --no-config. Fixed by temporarily enabling "Use Curl and Wget Configuration Files"

@roblourens My first attempt to bypass that and use the "Local Server Download" setting did not work, because VS Code will look for scp.exe in the same path as my ssh.cmd. That does exist, but because the extension is .exe (and not .bat or .cmd), the {shell: true} parameter won't get passed, causing the EINVAL error again.

Hi sorry I've seen this far too late. You can just set a symlink to scp i think? There's a windows equivalent command that isn't a shortcut too but I can't remember what it is sorry. Might be PowerShell...