search

github / gh-net

A network bridge between a Codespace and a local machine.
https://github.com/github/gh-net
286 stars 22 forks source link

[nat]: nat records are freed up prematurely #17

Closed legomushroom closed 2 years ago

legomushroom commented 2 years ago

Recently used NAT records can be cleaned up as old. This causes long-open TCP sockets to be reset by kernel.

Reproduce steps

  1. Create a long-running TCP connection (~30-60 seconds).
  2. Send "data" messages around periodically.
  3. Notice that after some time the connection is closed by a RST message from kernel.

Expected behavior

Long-running TPC connections do not close unexpectedly.

Logs

```log 2022-05-06T20:30:58.759556Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream) 2022-05-06T20:30:58.759615Z TRACE client:interface:stream: to network: IP V4 (id: 0x7E33): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CC Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06D8 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC47B7 (4294723511) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC0 A8761BA6 Checksum:............................0x1BF8 (7160) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.759708Z TRACE client:interface:stream:sender: sending packet: IP V4 (id: 0x7E33): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CC Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06D8 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC47B7 (4294723511) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC0 A8761BA6 Checksum:............................0x1BF8 (7160) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.759834Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.765377Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.765426Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.765460Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.765482Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.765608Z TRACE client:interface:stream: in IP V4 (id: 0x7E34): Source:...............................172.16.5.4 Destination:.......................192.168.86.27 Checksum:.................................0xF4B7 Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................53696 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06D8 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC5307 (4294726407) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC4 A8761BAC Checksum:...........................0xC7FE (51198) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.765706Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream) 2022-05-06T20:30:58.765764Z TRACE client:interface:stream: to network: IP V4 (id: 0x7E34): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CB Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06D8 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC5307 (4294726407) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC4 A8761BAC Checksum:............................0x109E (4254) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.765859Z TRACE client:interface:stream:sender: sending packet: IP V4 (id: 0x7E34): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CB Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06D8 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC5307 (4294726407) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC4 A8761BAC Checksum:............................0x109E (4254) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.765990Z TRACE client:interface:stream: in IP V4 (id: 0x7E35): Source:...............................172.16.5.4 Destination:.......................192.168.86.27 Checksum:.................................0xF4B6 Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................53696 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC5 A8761BAC Checksum:...........................0xC7FE (51198) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.766001Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.847122Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream) 2022-05-06T20:30:58.847210Z TRACE client:interface:stream: to network: IP V4 (id: 0x7E35): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CA Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC5 A8761BAC Checksum:............................0x0BE4 (3044) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.847307Z TRACE client:interface:stream:sender: sending packet: IP V4 (id: 0x7E35): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26CA Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC5 A8761BAC Checksum:............................0x0BE4 (3044) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.847499Z TRACE client:interface:stream: in IP V4 (id: 0x7E36): Source:...............................172.16.5.4 Destination:.......................192.168.86.27 Checksum:.................................0xF4B5 Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................53696 -> 3000 Flags [0x11]:............................[ACK FIN] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC6 A8761BAC Checksum:...........................0xC7FE (51198) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.847618Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream) 2022-05-06T20:30:58.847686Z DEBUG client:interface:network: out IP V4 (id: 0x00): Source:............................192.168.86.27 Destination:..........................172.16.5.4 Checksum:.................................0x6E3C Flags:......................................0x02 Payload len:................................1232 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................3000 -> 53696 Flags [0x18]:............................[ACK PSH] Window:.....................................0x0808 Payload len:..................................1200 SEQ:.......................0xFFFC5307 (4294726407) ACK:.......................0xB7A38826 (3080947750) Expected ACK:..............0xFFFC57B7 (4294727607) Options: No operation: [] No operation: [] Timestamps: A8761C1D 38584AC4 Checksum:...........................0x3A7E (14974) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.847775Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream) 2022-05-06T20:30:58.847830Z TRACE client:interface:stream: to network: IP V4 (id: 0x7E36): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26C9 Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x11]:............................[ACK FIN] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC6 A8761BAC Checksum:............................0x0BE2 (3042) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.847917Z TRACE client:interface:stream:sender: sending packet: IP V4 (id: 0x7E36): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26C9 Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x11]:............................[ACK FIN] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38826 (3080947750) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38827 (3080947751) Options: No operation: [] No operation: [] Timestamps: 38584AC6 A8761BAC Checksum:............................0x0BE2 (3042) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.848014Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.848047Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.848078Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream) 2022-05-06T20:30:58.848129Z DEBUG client:interface:network: out IP V4 (id: 0x00): Source:............................192.168.86.27 Destination:..........................172.16.5.4 Checksum:.................................0x72EC Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................3000 -> 53696 Flags [0x10]:................................[ACK] Window:.....................................0x0808 Payload len:.....................................0 SEQ:.......................0xFFFC57B7 (4294727607) ACK:.......................0xB7A38827 (3080947751) Expected ACK:..............0xFFFC57B8 (4294727608) Options: No operation: [] No operation: [] Timestamps: A8761C1D 38584AC6 Checksum:...........................0xEA32 (59954) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.849922Z TRACE client:interface:nats:tcp:id: NAT handled (from upstream) 2022-05-06T20:30:58.850017Z DEBUG client:interface:network: out IP V4 (id: 0x00): Source:............................192.168.86.27 Destination:..........................172.16.5.4 Checksum:.................................0x72EC Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................3000 -> 53696 Flags [0x11]:............................[ACK FIN] Window:.....................................0x0808 Payload len:.....................................0 SEQ:.......................0xFFFC57B7 (4294727607) ACK:.......................0xB7A38827 (3080947751) Expected ACK:..............0xFFFC57B8 (4294727608) Options: No operation: [] No operation: [] Timestamps: A8761C1F 38584AC6 Checksum:...........................0xEA2F (59951) Data offset:...................................0x8 Urgent pointer:.............................0x0000 2022-05-06T20:30:58.852925Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.852969Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.852997Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.853017Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.875327Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.875378Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.875407Z TRACE client:interface:nats:tcp:id: overriding loopback source to original upstream IP address 2022-05-06T20:30:58.875428Z TRACE client:interface:nats:tcp:id: 🔙 NAT from upstream: not our packet 2022-05-06T20:30:58.875543Z TRACE client:interface:stream: in IP V4 (id: 0x7E37): Source:...............................172.16.5.4 Destination:.......................192.168.86.27 Checksum:.................................0xF4A8 Flags:......................................0x02 Payload len:..................................44 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................53696 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38827 (3080947751) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38828 (3080947752) Options: No operation: [] No operation: [] Timestamps: 38584B33 A8761C1D No operation: [] No operation: [] SACK: [FF, FC, 53, 07, FF, FC, 57, B7] Checksum:...........................0xC80A (51210) Data offset:...................................0xB Urgent pointer:.............................0x0000 2022-05-06T20:30:58.875635Z TRACE client:interface:nats:tcp:id: NAT handled (from downstream) 2022-05-06T20:30:58.875661Z TRACE client:interface:nats: removed NAT [tcp_172.16.5.4:192.168.86.27_53696:3000] 2022-05-06T20:30:58.875704Z TRACE client:interface:nats:tcp:firewall: dropping 2022-05-06T20:30:58.875723Z TRACE client:interface:nats:tcp:firewall: removing TCP drop rule 2022-05-06T20:30:58.876356Z TRACE client:interface:nats:tcp:firewall: rule removed 2022-05-06T20:30:58.876392Z TRACE client:interface:nats:tcp:firewall: dropped 2022-05-06T20:30:58.876482Z TRACE client:interface:stream: to network: IP V4 (id: 0x7E37): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26BC Flags:......................................0x02 Payload len:..................................44 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38827 (3080947751) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38828 (3080947752) Options: No operation: [] No operation: [] Timestamps: 38584B33 A8761C1D No operation: [] No operation: [] SACK: [FF, FC, 53, 07, FF, FC, 57, B7] Checksum:...........................0x2A34 (10804) Data offset:...................................0xB Urgent pointer:.............................0x0000 2022-05-06T20:30:58.876589Z TRACE client:interface:stream:sender: sending packet: IP V4 (id: 0x7E37): Source:................................127.0.0.1 Destination:.......................192.168.86.27 Checksum:.................................0x26BC Flags:......................................0x02 Payload len:..................................44 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................58300 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38827 (3080947751) ACK:.......................0xFFFC57B7 (4294727607) Expected ACK:..............0xB7A38828 (3080947752) Options: No operation: [] No operation: [] Timestamps: 38584B33 A8761C1D No operation: [] No operation: [] SACK: [FF, FC, 53, 07, FF, FC, 57, B7] Checksum:...........................0x2A34 (10804) Data offset:...................................0xB Urgent pointer:.............................0x0000 2022-05-06T20:30:58.876723Z TRACE client:interface:stream: in IP V4 (id: 0x00): Source:...............................172.16.5.4 Destination:.......................192.168.86.27 Checksum:.................................0x72EC Flags:......................................0x02 Payload len:..................................32 DSCP:.......................................0x00 ECN:........................................0x00 TCP: Ports:...............................53696 -> 3000 Flags [0x10]:................................[ACK] Window:.....................................0x06E1 Payload len:.....................................0 SEQ:.......................0xB7A38827 (3080947751) ACK:.......................0xFFFC57B8 (4294727608) Expected ACK:..............0xB7A38828 (3080947752) Options: No operation: [] No operation: [] Timestamps: 38584B34 A8761C1F Checksum:...........................0xEAE8 (60136) Data offset:...................................0x8 Urgent pointer:.............................0x0000 ```
legomushroom commented 2 years ago

Fixed in v0.6.4 🎉