VSCode Remote SSH Support

[mikeurbach]

I am hoping to use the HGDB VSCode extension to debug Chisel programs remotely. The setup I'm talking about is using the Visual Studio Code Remote - SSH setup. When I am connected, it looks like this:

I would like to be able to open the files on the remote host in my editor for debugging, and run the simulator on the remote host. This is common for big hardware projects, where the codebase is normally hosted and built on a remote machine, and the commercial simulators are also hosted remotely.

When I try to start up the HGDB debug session, I encounter this error:

I have tried this with two different configurations in my launch.json.

First, I tried "runtimeIP": "127.0.0.1", "runtimePort": 8888. I was thinking the extension runs on the remote host, so I tried connecting to localhost (meaning remote in this case), and the port the runtime is listening on (8888 by default). This yielded the error above.

Second, I thought maybe the extension runs on my host, so I set up an SSH tunnel from my localhost's port 8080 to the remote host's port 8888, and tried to configure launch.json with "runtimeIP": "127.0.0.1", "runtimePort": 8888. This also yielded the above error.

Finally, I tried not using the VSCode Remote SSH capability, and opened the file locally. I still ran the simulator on the remote host, and used an SSH tunnel as mentioned above to connect. This does work, so the issue seems to be something related to using the Remote SSH capability. Unfortunately, Remote SSH is the normal way to use VSCode on my team, so this is something we'd like to get working.

Please let me know if I can provide more information or help debug. I was reading a bit about how VSCode extensions and remote support work, and I found this interesting tidbit here:

VS Code does not itself support remote debugging: this is a feature of the debug extension you are using, and you should consult the extension's page in the Marketplace for support and details.

There is, however, one exception: the Node.js debugger included in VS Code supports remote debugging. See the Node.js Debugging topic to learn how to configure this.

It sounds to me that this is something the extension needs to take care of. Perhaps we can take a look at how the Node.js debugger achieves remote debugging, and port that to hgdb-debugger.

[Kuree]

Hi @mikeurbach, thanks for providing detailed steps.

I was not able to reproduce the error. See the screenshot below, where kiwi is a remote server in my group.

It sounds to me that this is something the extension needs to take care of. Perhaps we can take a look at how the Node.js debugger achieves remote debugging, and port that to hgdb-debugger.

hgdb technically only supports remote debugging, since the runtime library opens up a TCP port for debugger connections. So I don't think it's an issue here.

Since the error message is not thrown by the extension itself, I might have some ideas why the error is happening. VSCode uses debug adapter protocols, which itself uses RPC to communicate. You can see the code here: https://github.com/Kuree/hgdb-debugger/blob/a4f452a4ae517c610993ca0aef4ffbaa23044bdc/vscode/src/extension.ts#L115-L126

where it starts the debug adapter server with a random port. I believe this server runs on the remote (ssh server). Since your firewall blocks all ports other than 22, the local VSCode fails to connect to the debug adapter server.

This is, however, is my educated guess. To see if it's the issue, we need to replace the random port with a known port. If you think this makes sense, I can try to figure out a way to make this port configurable.

[mikeurbach]

it starts the debug adapter server with a random port. I believe this server runs on the remote (ssh server). Since your firewall blocks all ports other than 22, the local VSCode fails to connect to the debug adapter server.

Thanks for taking a look and sharing those details, I didn't realize it was starting a debug adapter server. That makes sense to me, I think you are right that I will not have access to the port the debug adapter server is listening on.

Being able to configure the port to listen on would be helpful, or even just logging which port was chosen would help. I think either way, I will need to use SSH to tunnel from my local to that port, so being able to do that reliably should solve this.

[Kuree]

I pushed a change that adds dapPort as a configuration in launch.json. I played around with it a little bit and it hasn't broken anything yet, but I don't have a firewalled remote server so I'm not 100% sure it's working. Can you try to install this VSCode package from the test? You can also build it from the source using npm run package.

[mikeurbach]

Awesome, thanks for the quick turnaround! I will try that out. If it needs any tweaks, I will try to test them out myself and submit a patch.

[mikeurbach]

I tried this out myself, but the simple steps I had in mind didn't work. I set the dapPort to something known (e.g. 6666), and set up an ssh tunnel to that port, like ssh -v -N -L 6666:<host>:6666 <host>, but I still receive the same "Cannot reconnect" error.

At this point, I'm trying to actually debug the situation. I'm reading up on how this debug adapter protocol works, and I'm trying to put some extra logging in the extension to see whats going on. So far, I've found that logging added right before Net.createServer is not showing up in the output (or else I'm logging the wrong way).

[Kuree]

So far, I've found that logging added right before Net.createServer is not showing up in the output (or else I'm logging the wrong way).

I tried that and it's showing up on my side.

What I did is to load the extension in the vscode, launch an extension debug session. This should bring up a new VSCode session called [Extension Development Host]. Starting an hgdb session in the new vscode code should allow you setting breakpoints in the original vscode.

Did you try to set breakpoint to see what's causing the issue?

[mikeurbach]

I will try debugging the extension itself, I haven't done that yet.

[Kuree]

Make sure you load the compiled version. Usually run the package command before debugging forces vscode to load the latest version.

Also you need to uninstall the marketplace version of hgdb-debugger. You have already done this but I just want to double check.

[mikeurbach]

Yep, thanks. I've been able to debug the extension, so I'll try to see what's going on.

[mikeurbach]

Strangely enough... when I run the extension in a debugger (on the remote host), I am able able to use it to debug my hardware on the remote machine. Perhaps debugging the extensions remotely is the difference? It sort of sounds like the extension was running locally when it failed (i.e. the debug adapter was on my local host, and failed to connect to the debugger port on the remote).

Looking at this diagram: https://code.visualstudio.com/api/advanced-topics/remote-extensions#architecture-and-extension-kinds, I think we want to make sure this is a "workspace" extension that runs remotely. I guess I can debug this: https://code.visualstudio.com/api/advanced-topics/remote-extensions#incorrect-execution-location.

EDIT: I confirmed it runs as a workspace extension on the remote host. I tried running it locally to see what happens, and it fails with the same error.

[Kuree]

Thanks for the update! I will add "extensionKind": ["workspace"] to the package declaration.

[mikeurbach]

Thanks, I think that will be helpful, but it still didn't solve my issue.

I also took a look at this: https://code.visualstudio.com/api/advanced-topics/remote-extensions#forwarding-localhost. That sounded like what I was missing, so I tried it out like this: https://github.com/Kuree/hgdb-debugger/compare/master...mikeurbach:remote-ssh?expand=1.

That was nice, because it automatically set up a port forwarded to where the debug adapter server was started:

However, I don't think this is materially any different than when I manually port forwarded with SSH earlier today. I still face the same "Cannot reconnect" error.

At this point I am not sure what else to try. I don't know why it worked on the kiwi machine you tested with, but not what I'm using. Perhaps there are differences in the network configuration that confuse VSCode in my case.

[Kuree]

I will try to set up a VM that only has 22 port open and see if I can reproduce the error. I will keep you updated.

[mikeurbach]

Awesome! Thanks for helping debug this scenario.

There is one more data point I should share in case its relevant: I actually don't connect to my remote host directly, it is within an internal network, and I SSH through a bastion host. This is normally handled transparently with my ~/.ssh/config, so when I tell VSCode to connect to the remote host directly, VSCode hops through the bastion no problem. But perhaps this setup confuses the extension networking?

Hopefully you can at least reproduce with a simple setup using one remote VM without getting into bastions and more complex configurations. As always, let me know if I can help at all, and I will keep trying on my end.

[Kuree]

Here is what I did that seems to work:

1. Set up a virtualbox VM that only has 22 port forwarded (using standard NAT).
2. Remove hgdb-debugger from local VSCode
3. Launch remote ssh from VSCode (this should bring up a new VSCode window)
4. Download the latest vsix from action page and scp it to the VM
5. Install the vsix package directly in the remote host (see the screenshot)
6. Launch the debugging session as usual after starting the simulator in a different terminal

Please let me know if that works for you.

[mikeurbach]

It still does not work with the remote machine I've been testing. I will try to reproduce with a VM to make sure I can get that to work at least.

[Kuree]

@mikeurbach Any update on this? I'm suspecting there is a complex NAT rules in your server. One potential solution is to use WebRTC with STUN server, which in theory should work with your network configuration.

[mikeurbach]

No update, I have been pulled off into other tasks. I will be testing it again this week without worrying about the remote configuration. Once I have some demo working I will figure out the best way to make remote work for our setup. WebRTC and STUN is an interesting idea, I'll definitely consider that.

[Kuree]

Another option would be using ngrok, which is often used to do NAT hole punching. You can see the example here: https://ngrok.com/docs#tcp-examples

It worked for my remote setup, i.e. running ngrok on the server and tunnel through. However, the server doesn't have any NAT or firewall rules so I'm not sure it applies to your case. Please let me know if it works.

[mikeurbach]

Thanks for the pointer, I will take a look at ngrok as well. Before putting any more time into it, I need to learn about how these private remote machines are actually configured.