Language Server (Pyright/Ruff-lsp) fails to connect over Emacs TRAMP (SSH)

[jleechpe]

What happened?

When trying to connect to a language server within the DevPod from Emacs over Tramp it fails after a 30 second timeout. Connecting to the devpod through via the docker: protocol rather than ssh: everything works as expected. ssh: protocol also works when connecting to language servers that don't have the devpod-cli wrapper in the ProxyCommand

[jsonrpc] D[08:38:16.164] Running language server: /bin/sh -c stty raw > /dev/null; .venv/bin/pyright-langserver --stdio
[jsonrpc] e[08:38:16.168] --> initialize[1] {"jsonrpc":"2.0","id":1,"method":"initialize","params":{"processId":null,"clientInfo":{"name":"Eglot","version":"1.17"},"rootPath":"/workspaces/repo/","rootUri":"file:///workspaces/repo","initializationOptions":{},"capabilities":{"workspace":{"applyEdit":true,"executeCommand":{"dynamicRegistration":false},"workspaceEdit":{"documentChanges":true},"didChangeWatchedFiles":{"dynamicRegistration":false},"symbol":{"dynamicRegistration":false},"configuration":true,"workspaceFolders":true},"textDocument":{"synchronization":{"dynamicRegistration":false,"willSave":true,"willSaveWaitUntil":true,"didSave":true},"completion":{"dynamicRegistration":false,"completionItem":{"snippetSupport":true,"deprecatedSupport":true,"resolveSupport":{"properties":["documentation","details","additionalTextEdits"]},"tagSupport":{"valueSet":[1]}},"contextSupport":true},"hover":{"dynamicRegistration":false,"contentFormat":["markdown","plaintext"]},"signatureHelp":{"dynamicRegistration":false,"signatureInformation":{"parameterInformation":{"labelOffsetSupport":true},"documentationFormat":["markdown","plaintext"],"activeParameterSupport":true}},"references":{"dynamicRegistration":false},"definition":{"dynamicRegistration":false,"linkSupport":true},"declaration":{"dynamicRegistration":false,"linkSupport":true},"implementation":{"dynamicRegistration":false,"linkSupport":true},"typeDefinition":{"dynamicRegistration":false,"linkSupport":true},"documentSymbol":{"dynamicRegistration":false,"hierarchicalDocumentSymbolSupport":true,"symbolKind":{"valueSet":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26]}},"documentHighlight":{"dynamicRegistration":false},"codeAction":{"dynamicRegistration":false,"resolveSupport":{"properties":["edit","command"]},"dataSupport":true,"codeActionLiteralSupport":{"codeActionKind":{"valueSet":["quickfix","refactor","refactor.extract","refactor.inline","refactor.rewrite","source","source.organizeImports"]}},"isPreferredSupport":true},"formatting":{"dynamicRegistration":false},"rangeFormatting":{"dynamicRegistration":false},"rename":{"dynamicRegistration":false},"inlayHint":{"dynamicRegistration":false},"publishDiagnostics":{"relatedInformation":false,"codeDescriptionSupport":false,"tagSupport":{"valueSet":[1,2]}}},"window":{"showDocument":{"support":true},"workDoneProgress":true},"general":{"positionEncodings":["utf-32","utf-8","utf-16"]},"experimental":{}},"workspaceFolders":[{"uri":"file:///workspaces/repo","name":"/ssh:aurelius.devpod:/workspaces/repo/"}]}}

[jsonrpc] i[08:38:46.170] [1] timed-out request ':initialize'
[jsonrpc] D[08:38:46.271] Connection state change: `killed
'

----------b---y---e---b---y---e----------

What did you expect to happen instead?

Language Server initializes and connects successfully.

How can we reproduce the bug? (as minimally and precisely as possible)

Start a devpod container, connect to code files within it using TRAMP ( /ssh:name.devpod:/path/to/files or /sshx:name.devpod:/path/to/files) then try to start a language server ( M-x eglot [then provide path to Language Server executable if it fails to find]). ... Eglot (language server handler) will successfully set up the LSP connection then timeout ~30 seconds later.

Do the same but connect via docker (/docker:<container-name>:/path/to-files) and Eglot will successfully connect and interact.

Local Environment:

• DevPod Version: 0.5.16
• Operating System: linux
• ARCH of the OS: AMD64

DevPod Provider:

• Local/remote provider: docker

Anything else we need to know?

[pascalbreuninger]

Hi @jleechpe, thanks for opening the issue. I'm personally not an emacs user and only understood half of the issue description.

Could you isolate the problem in the ssh connection somehow?

If not I'd be grateful for any help from the community here

[jleechpe]

I tried using the debug flag both in TRAMP and in the devpod ssh connection and didn't see very much that helped hint at what might be going wrong.

09:52:53.745464 tramp-send-command (6) # cd /workspaces/repo/ &&  exec  2>'/tmp/tramp.aZI9TA' env INSIDE_EMACS\=31.0.50\,tramp\:2.7.1 PS1\=/ssh\:repo.devpod\:/workspaces/repo/\ \#\$\  /bin/sh -c stty\ raw\ \>\ /dev/null\;\ rye\ run\ pyright-langserver\ --stdio
09:52:53.746626 tramp-send-command (6) # (if test -h "/workspaces/repo/"; then echo t; else echo nil; fi) && \readlink --canonicalize-missing /workspaces/repo/ 2>/dev/null; echo tramp_exit_status $?
09:52:53.747891 tramp-wait-for-regexp (6) #
nil
/workspaces/repo
tramp_exit_status 0

(SSH to another machine for the same command has the equivalent output accounting for path but then has subsequent commands that occur afterwards)

The flow that Emacs is trying to use is:

• Tramp opens the file over SSH to display locally but makes any changes/lockfiles on the remote filesystem (ideally using some form of SSH ControlMaster to reuse tunnels but I tested both with and without any settings for ControlMaster)
• Eglot (LSP) launches the Language Server based on the remote path/project/applicable context (virtualenv doesn't load/parse properly in Tramp so have to specify the path or use a wrapper on the remote host)
• Eglot uses JSONRPC to talk with the LSP over the SSH tunnel.

From what I can see happening...

• When connecting over Docker to a Devpod host [using docker ps to find the container first] (uses docker exec ... rather than ssh ... as far as I can tell) the Language Server initializes properly and responds to the JSONRPC init command
• When connecting over SSH to a non-DevPod host JSONRPC responds properly
• When connecting to a DevPod over SSH the various echo/generic calls respond properly (TRAMP testing connection works) but the JSONRPC response times out rather than ever returning.

[jleechpe]

@pascalbreuninger Looking back into this again. Is there a way to connect to the devpod via SSH without going through the devpod-cli ssh wrapper/proxy command (even if it is not recommended/inconvenient outside of debugging)? The fact that it works when connecting to it as a docker container (and ssh working for non-devpod hosts) makes me think there's something not being passed through the proxy properly (It expects a JSONRPC sentinel process to communicate with that never seems to establish)

[pascalbreuninger]

@jleechpe not directly. If you need to split it up, you could start the workspace with DevPod and then docker exec into the container once everything is set up. That's a bad experience though and I'd rather fix the underlying issue

[jleechpe]

@pascalbreuninger That's effectively what I was doing to troubleshoot.

• If I connect using ssh (devpod-cli ssh due to .ssh/config) then the pyright langserver times out after 30 seconds since it never gets a jsonrpc response.
• If I connect using docker exec then it works as expected
• If I connect to a non-devpod host using ssh (2nd local machine where I cloned the same code repo) then it works as expected

[jleechpe]

Was able to get normal SSH working by updating the password for codespace (which I'm assuming will break everything else but was enough to use normal SSH commands for testing.

Switching to just the normal SSH connection ( ssh codespace@172.17.0.2 -p 2222) everything works as expected. So it looks like the issue is somewhere in how the Go SSH implementation is setting up the tunnel or wrapping the messages that prevent the JSONRPC connection from establishing properly.