Help debugging issue with port-forwarding

[cirego]

Hey @DeCarabas, hope you're doing well! I'm using fwd here at new gig and I'm running into an issue where a port isn't being forwarded by fwd.

Running lsof, here are two ports with active listeners:

coder@dev-chris:~$ sudo lsof -i:8250
COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
vault   17947 coder    9u  IPv4  26541      0t0  TCP localhost:8250 (LISTEN)
coder@dev-chris:~$ sudo lsof -i:6060
COMMAND  PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
coder   1782 coder    7u  IPv4  38156      0t0  TCP localhost:6060 (LISTEN)

6060 is forwarded, 8250 is not.

Without going too much further into the code, are there additional debugging steps that you would recommend I take to debug this? I've tried turning on logs but nothing shows up.

Running fwd with -s does port-forward 8250. Here are the debug logs:

[CLIENT] Port config PortDesc { port: 8250, desc: "vault login -method=oidc --format=json -address=https://sitetest3-vault.simulpong.com:8200" } -> PortConfig { enabled: true, description: None }                                                                                                                                                                                    │
│[CLIENT] Starting port 8250 to 58311

[quodlibetor]

you could add -vvv at the port forwarding code, although I suppose I'm not actually certain that that output shows up in logs.

[DeCarabas]

Hmm, I'm not sure. If it works with -s but doesn't work otherwise then it's something to do with permissions reading procfs, which means there's something about the way that vault is being run that doesn't let your normal user poke at it. I don't know anything about your case though. (Or vault? Is that the HashiCorp vault?)

I despair of using logging for this, only because (a) the logging will be very verbose and (b) we don't have a good way of going through so much logging. Maybe using strace on fwd on the server side would help identify what isn't working? I expect you would just see open fail with EACCES though, which is just not that useful. :(

I guess: how is the process in question being run?

[cirego]

Thank you both for the responses! Is there a way to run the server in debug mode so that I can add logging? I'm building my own version of the server but adding logging seems to make the client hang :)

[DeCarabas]

Is there a way to run the server in debug mode so that I can add logging?

Make sure the debug version is the one on your path on the server. The client just runs ssh <whatever> fwd --server, so ssh is going to find the one in your path...

[DeCarabas]

but adding logging seems to make the client hang

This is concerning and I would be happy to debug this too.

[DeCarabas]

One other thing you might try: although not documented, fwd supports a certain amount of manual configuration. If the port for vault is known, then you could try writing a configuration file named ~/.fwd which contains the following TOML:

[servers.dev-chris.ports]
8250 = { enabled=true, description="vault" }

(Replace dev-chris with the name of the server you use with fwd, and 8250 with the port number.)

This doesn't work very well before 0ad0fb1, but I fixed it up recently. Maybe that's a workaround, if you can't figure it out otherwise?

[DeCarabas]

I've been thinking about this and as of a4745c9 I changed the policy about what ports get reported from the server to the client.

Now, if you're running a sufficiently new client, the client will report all ports from the server, even if it cannot identify the process that is listening on the port (say, because of permission issues).

These "anonymous" ports will be disabled by default, for security reasons, but they can be manually enabled with the UI.

(If this is too onerous, the configuration solution above still works.)

[cirego]

I've been thinking about this and as of a4745c9 I changed the policy about what ports get reported from the server to the client.

Now, if you're running a sufficiently new client, the client will report all ports from the server, even if it cannot identify the process that is listening on the port (say, because of permission issues).

These "anonymous" ports will be disabled by default, for security reasons, but they can be manually enabled with the UI.

(If this is too onerous, the configuration solution above still works.)

Thank you! I will give the new client a try!