All gateway traffic is unroutable

jwilbur-godaddy commented 2 years ago

Describe the bug

In the TUI, it appears that gh net manages to resolve DNS names, but it fails to make any of them routable. I see no NAT rules appear in my WSL 2 environment. My team also tried this on Mac OS and got NAT rules to appear, but still, no traffic was routable. I stopped sshd from codespaces and ran sudo /usr/sbin/sshd -dD to get debugging logs, which can be seen below:

debug1: sshd version OpenSSH_8.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: private host key #0: ssh-rsa SHA256:k9VYtTGN4jUIzHj/EKq+6y6uJisj1iY6g9lsVROToaI
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:ocBy0gcEdSuQoID0LBS3k/g5Q2ph6xyz80/Lh8hdkNs
debug1: private host key #2: ssh-ed25519 SHA256:ba71WhdgVVnBeAfR1y1Tk/iY6czrEsZXzyengW4ESfE
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-dD'
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 2222 on 0.0.0.0.
Server listening on 0.0.0.0 port 2222.
debug1: Bind to port 2222 on ::.
Server listening on :: port 2222.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: sshd version OpenSSH_8.2, OpenSSL 1.1.1f  31 Mar 2020
debug1: private host key #0: ssh-rsa SHA256:k9VYtTGN4jUIzHj/EKq+6y6uJisj1iY6g9lsVROToaI
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:ocBy0gcEdSuQoID0LBS3k/g5Q2ph6xyz80/Lh8hdkNs
debug1: private host key #2: ssh-ed25519 SHA256:ba71WhdgVVnBeAfR1y1Tk/iY6czrEsZXzyengW4ESfE
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 39600 on 127.0.0.1 port 2222 rdomain ""
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.4 pat OpenSSH* compat 0x04000000
debug1: permanently_set_uid: 105/65534 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com [preauth]
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: zlib@openssh.com [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug1: rekey out after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: Sending SSH2_MSG_EXT_INFO [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug1: rekey in after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user codespace service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug1: PAM: initializing for "codespace"
debug1: PAM: setting PAM_RHOST to "127.0.0.1"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user codespace service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug1: userauth_pubkey: test pkalg ssh-ed25519 pkblob ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /home/codespace/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: /home/codespace/.ssh/authorized_keys:3: matching key found: ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY
debug1: /home/codespace/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Accepted key ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY found at /home/codespace/.ssh/authorized_keys:3
debug1: restore_uid: 0/0
Postponed publickey for codespace from 127.0.0.1 port 39600 ssh2 [preauth]
debug1: userauth-request for user codespace service ssh-connection method publickey [preauth]
debug1: attempt 2 failures 0 [preauth]
debug1: temporarily_use_uid: 1000/1000 (e=0/0)
debug1: trying public key file /home/codespace/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: /home/codespace/.ssh/authorized_keys:3: matching key found: ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY
debug1: /home/codespace/.ssh/authorized_keys:3: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Accepted key ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY found at /home/codespace/.ssh/authorized_keys:3
debug1: restore_uid: 0/0
debug1: auth_activate_options: setting new authentication options
debug1: do_pam_account: called
Accepted publickey for codespace from 127.0.0.1 port 39600 ssh2: ED25519 SHA256:oJusw5Gxu9cT+lVe3ggOqrEyyRuxSe9zFuEzKQMHxpY
debug1: monitor_child_preauth: codespace has been authenticated by privileged process
debug1: auth_activate_options: setting new authentication options [preauth]
debug1: Enabling compression at level 6. [preauth]
debug1: monitor_read_log: child log fd closed
debug1: PAM: establishing credentials
User child is on pid 26161
debug1: SELinux support disabled
debug1: PAM: establishing credentials
debug1: permanently_set_uid: 1000/1000
debug1: rekey in after 134217728 blocks
debug1: rekey out after 134217728 blocks
debug1: ssh_packet_set_postauth: called
debug1: Enabling compression at level 6.
debug1: active: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch
debug1: server_input_channel_open: ctype session rchan 0 win 2097152 max 32768
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype no-more-sessions@openssh.com want_reply 0
debug1: server_input_channel_req: channel 0 request env reply 0
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req env
debug1: server_input_channel_req: channel 0 request exec reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req exec
Starting session: command for codespace from 127.0.0.1 port 39600 id 0
 Connection closed by 127.0.0.1 port 39600
debug1: channel 0: free: server-session, nchannels 1
Close session: user codespace from 127.0.0.1 port 39600 id 0
debug1: do_cleanup
debug1: temporarily_use_uid: 1000/1000 (e=1000/1000)
debug1: restore_uid: (unprivileged)
Transferred: sent 4720, received 3336 bytes
Closing connection to 127.0.0.1 port 39600
debug1: compress outgoing: raw data 6410, compressed 2487, factor 0.39
debug1: compress incoming: raw data 5590, compressed 1152, factor 0.21
debug1: PAM: cleanup
debug1: PAM: closing session
debug1: PAM: deleting credentials

The logs seemed to hang on the line starting with "Starting session."

Here are the logs on the client side:

2022-05-23T15:19:17.805020Z  INFO start-command: app:                                                                                                                                                  version: "0.6.4"                                                                                                                                                                                       sha: "8b282b2f1d52abb127ccfb7eca1d04f4344af4df"                                                                                                                                                        built: "Wed, 11 May 2022 23:32:02 +0000"                                                                                                                                                               profile: "release"                                                                                                                                                                                     os: "linux"                                                                                                                                                                                            family: "unix"                                                                                                                                                                                         arch: "x86_64"                                                                                                                                                                                         endian: "little"                                                                                                                                                                                       cores: 16                                                                                                                                                                                              pointer-width: "64"                                                                                                                                                                                    debug: false                                                                                                                                                                                           in codespace: "false"                                                                                                                                                                                  2022-05-23T15:19:17.805085Z  INFO start-command: start command:                                                                                                                                        --dns: "true"                                                                                                                                                                                          --gui: "true"                                                                                                                                                                                          --repo: "github/gh-net"                                                                                                                                                                                --trace: "info"                                                                                                                                                                                        --location: "local"                                                                                                                                                                                    --telemetry: "true"                                                                                                                                                                                    2022-05-23T15:19:19.379916Z  INFO local: run_local is_dns: "true"                                                                                                                                      2022-05-23T15:19:19.379966Z  INFO local: codespace-name: "jwilbur-godaddy-gdcorp-im-account-billing-4j9vrx5pvcjv5j"                                                                                    2022-05-23T15:19:19.383595Z  INFO client: suitable network interfaces: ["lo", "eth0"]                                                                                                                  2022-05-23T15:19:19.383670Z  INFO client: starting 2 network interface jobs                                                                                                                            2022-05-23T15:19:19.383760Z  INFO network: copy queue to stream job started                                                                                                                            2022-05-23T15:19:19.383861Z  INFO network: connected "eth0"                                                                                                                                            2022-05-23T15:19:19.383864Z  INFO stream: connected "eth0"                                                                                                                                             2022-05-23T15:19:19.419967Z  INFO network: connected "lo"                                                                                                                                              2022-05-23T15:19:19.420067Z  INFO stream: connected "lo"                                                                                                                                               2022-05-23T15:19:19.489895Z  INFO stream: copy stream to queue job started                                                                                                                             2022-05-23T15:20:23.859884Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("vortex.data.microsoft.com"), prefer_unicast: false, qtype: A, qclass: IN }]                   2022-05-23T15:20:23.924659Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("vortex.data.microsoft.com"), prefer_unicast: false, qtype: AAAA, qclass: IN }]                2022-05-23T15:20:23.977810Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:20:43.931318Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("mobile.events.data.microsoft.com"), prefer_unicast: false, qtype: A, qclass: IN }]            2022-05-23T15:20:43.966790Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("mobile.events.data.microsoft.com"), prefer_unicast: false, qtype: AAAA, qclass: IN }]         2022-05-23T15:20:44.064049Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:21:02.710184Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("az764295.vo.msecnd.net"), prefer_unicast: false, qtype: A, qclass: IN }]                      2022-05-23T15:21:02.734576Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("az764295.vo.msecnd.net"), prefer_unicast: false, qtype: AAAA, qclass: IN }]                   2022-05-23T15:21:02.778119Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:21:06.551310Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("dc.services.visualstudio.com"), prefer_unicast: false, qtype: A, qclass: IN }]                2022-05-23T15:21:06.649880Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("dc.services.visualstudio.com"), prefer_unicast: false, qtype: AAAA, qclass: IN }]             2022-05-23T15:21:06.704470Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:21:06.706772Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("json.schemastore.org"), prefer_unicast: false, qtype: A, qclass: IN }]                        2022-05-23T15:21:06.768657Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("json.schemastore.org"), prefer_unicast: false, qtype: AAAA, qclass: IN }]                     2022-05-23T15:21:06.835240Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:25:49.921811Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("default.exp-tas.com"), prefer_unicast: false, qtype: A, qclass: IN }]                         2022-05-23T15:25:49.945819Z  INFO dns-resolver-sink: DNS hostname solicitation: [Question { qname: Name("default.exp-tas.com"), prefer_unicast: false, qtype: AAAA, qclass: IN }]                      2022-05-23T15:25:49.978747Z  INFO dns-resolver-sink: no records, adding NXDOMAIN responses                                                                                                             2022-05-23T15:36:19.085715Z  WARN stream: no packets left                                                                                                                                              2022-05-23T15:36:19.085715Z  WARN stream: no packets left

Reproduce steps

On Windows 10, connect to GlobalProtect VPN.
Start WSL 2.
Run gh net start.
Observe that no NAT rules appear in the list. Hosts on the same network as the client machine (Windows 10 / WSL 2) cannot be reached.

Expected behavior

I want to be able to reach a host on my local network from Codespaces.

Desktop (please complete the following information):

OS: Windows 10 / WSL 2
Version: 10.0.19042 N/A Build 19042
Platform architecture: Intel64 Family 6 Model 141 Stepping 1 GenuineIntel ~2496 Mhz

Logs Please attach logs to created issue. The best way for getting logs is to use VSCode client: connect to a Codespace and run > Codespaces: Export Logs command in command palette. Please note a logs file name shown in the UI. logs.zip

jwilbur-godaddy commented 2 years ago

Here are logs from my coworker using Mac OS:

2022-05-23T16:57:48.047256Z  INFO start-command: app:
version: "0.6.4"
sha: "8b282b2f1d52abb127ccfb7eca1d04f4344af4df"
built: "Wed, 11 May 2022 23:31:41 +0000"
profile: "release"
os: "macos"
family: "unix"
arch: "x86_64"
endian: "little"
cores: 16
pointer-width: "64"
debug: false
in codespace: "false"
2022-05-23T16:57:48.047393Z  INFO start-command: start command:
--dns: "true"
--gui: "true"
--repo: "github/gh-net"
--trace: "info"
--location: "local"
--telemetry: "true"
2022-05-23T16:57:49.958419Z  INFO local: run_local is_dns: "true"
2022-05-23T16:57:49.958472Z  INFO local: codespace-name: "dnguyen1-godaddy-gdcorp-im-account-billing-pjpppw4pwcrv9g"
2022-05-23T16:57:49.961431Z  INFO client: suitable network interfaces: ["lo0", "en0", "utun2"]
2022-05-23T16:57:49.961661Z  INFO client: starting 3 network interface jobs
2022-05-23T16:57:49.961810Z  INFO network: copy queue to stream job started
2022-05-23T16:57:49.962367Z  INFO stream: connected "lo0"
2022-05-23T16:57:49.962464Z  INFO network: connected "en0"
2022-05-23T16:57:49.962456Z  INFO network: connected "lo0"
2022-05-23T16:57:49.962549Z  INFO stream: connected "en0"
2022-05-23T16:57:49.962756Z  INFO network: connected "utun2"
2022-05-23T16:57:49.962765Z  INFO stream: connected "utun2"
2022-05-23T16:57:52.966516Z  INFO stream: copy stream to queue job started
2022-05-23T16:58:53.295229Z ERROR stream: error: [Os { code: 65, kind: HostUnreachable, message: "No route to host" }]
2022-05-23T16:58:55.314072Z  WARN stream: Cannot send IpPacket to the interface queue for NetworkInterface { name: "en0", description: "", index: 6, mac: Some(3c:22:fb:1a:1f:d1), ips: [V4(Ipv4Network { addr: 10.0.0.129, prefix: 24 })], flags: 34915 }, Closed(FromStreamPacket { packet:

legomushroom commented 2 years ago

@jwilbur-godaddy I never tested it under WSL but definitely want to make it work. Mind starting the extension with trace tracing level [sudo] gh net start --trace trace, repro the issue and send me the logs? The logs might contain personal data, so please DM them over to legomushroom@github.com if you have concerns attaching the logs here. Thanks! 🙏

legomushroom commented 2 years ago

@jwilbur-godaddy just a note:

The NAT rules are created dynamically when a request is routed. Hence the list will be empty until you will try to make a request, e.g. wget -d hrweb -4 or similar.
For successfully completed(closed TCP connect) requests, the NAT record is immediately removed from the list since the NAT record is not needed anymore. So if you do wget -d hrweb which quickly downloads the page and closes underlying TCP connection, you might not notice the NAT list change at all(gui has a rendering delay of 1s for perf reasons).

legomushroom commented 2 years ago

@jwilbur-godaddy I have a potential fix for this on the legomushroom/gh-net repo, mind trying it out?

gh extension remove net
gh extension install legomushroom/gh-net
sudo gh net -V # should print `gh net 0.6.7`

Then use the extension as before, but please add the --trace trace argument so it would have additional debugging details in case it still has some issues.

Thanks! 🙏

legomushroom commented 2 years ago

The fixed mentioned is released in 0.8.3 🎉 Please give it a try ☺️

iMicknl commented 2 years ago

@jwilbur-godaddy did 0.8.3 fix your issues regarding using gh net in WSL2? I have a similar issue in WSL2 where my DNS seems to be working, but my NAT stays empty.

I cannot use wget / ping from my Codespace to a local address, thus it seems it is not just the NAT panel which doesn't render, but the whole NAT functionality does not seem to be working.

Any debugging / logs I can provide for this? @legomushroom

jwilbur-godaddy commented 1 year ago

I never actually confirmed that it worked. When I got back from paternity leave, I tried again, but GitHub codespaces itself wasn't working. I couldn't connect to them over SSH. I don't think it was the fault of this extension. What I can say is that it seems like I got further along. It seems like this issue would be fixed for me if I could get Codespaces to work in the first place. Thank you!

mbroadhead-nr commented 10 months ago

@legomushroom I'm trying to connect from WSL and am also getting nothing listed in the left NAT panel

2023-10-15T19:02:24.356084Z  INFO start-command: app:
version: "0.12.4"
sha: "0514431be26c0fa4a541a134be39178844d19ecc"
built: "Thu, 15 Sep 2022 21:03:46 +0000"
profile: "release"
os: "linux"
family: "unix"
arch: "x86_64"
endian: "little"
cores: 16
pointer-width: "64"
debug: false
in-codespace: "false"
2023-10-15T19:02:24.356208Z  INFO start-command: start command:
--dns: "true"
--gui: "true"
--repo: "github/gh-net"
--trace: "info"
--trace-dest: None
--location: "local"
--telemetry: "true"
--codespace: None
2023-10-15T19:02:26.136263Z  INFO local: codespace-name: "xxxxxxxx"
2023-10-15T19:02:26.139405Z  INFO client: target network interfaces:
lo:
  name: "lo"
  description: ""
  ips: [V4(Ipv4Network { addr: 127.0.0.1, prefix: 8 }), V6(Ipv6Network { addr: ::1, prefix: 128 })]
  mac: Some(00:00:00:00:00:00)
  up: true
  loopback: true
  point-to-point: false
  default gateway: false
eth0:
  name: "eth0"
  description: ""
  ips: [V4(Ipv4Network { addr: xxxxxx, prefix: 20 }), V6(Ipv6Network { addr: xxxxxx, prefix: 64 })]
  mac: Some(00:xxxxxx)
  up: true
  loopback: false
  point-to-point: false
  default gateway: true
2023-10-15T19:02:26.139611Z  INFO client: starting 2 network interface jobs
2023-10-15T19:02:26.139994Z  INFO stream: connected "lo"
2023-10-15T19:02:26.140310Z  INFO stream: connected "eth0"
2023-10-15T19:02:26.140793Z  INFO network: copy queue to stream job started
2023-10-15T19:02:26.140811Z  INFO stream: copy stream to queue job started
2023-10-15T19:02:26.183524Z  INFO network: connected ""
2023-10-15T19:02:26.184946Z  INFO network: connected ""
...
2023-10-15T19:02:35.382335Z  INFO resovle-job: no records, adding NXDOMAIN responses

github / gh-net

All gateway traffic is unroutable #25