WheezyE
commented
2 years ago Thank you to KM4ACK & WH6AZ (of iOS RadioMail) for finding the root cause of this issue!
Just consolidating some notes & tests here on this github ticket so we can work in the open.
The VARA TCP re-connection problem
- Linux TCP ports enter a
TIME_WAITstate after the last connection on them is terminated. Apparently to prevent DoS attacks & also to prevent packet loss in some edge cases (references: 1, 2, 3) - The state of each TCP port can be viewed with
netstat | grep tcp. VARA's localhost:8300/localhost:8301 ports enterESTABLISHEDstate when an RMS Express VARA HF P2P session is first opened. Then, as soon as VARA HF is closed, the 8300/8301 ports enterTIME_WAITstate for Linux OS's. I timed how long the ports stay inTIME_WAITto be about 60 seconds. Then the ports disappear from netstat (they close & can be re-used again). - While ports 8300/8301 are in
TIME_WAIT, VARA HF will currently not attempt to connect to them. We believe that this is the cause of our issue. Said another way: If VARA is run, then VARA is closed, then VARA is run again, VARA will not re-establish a connection to RMS Express (or any other controller program over TCP) within this 60-second window.
Click to expand: Linux port states while VARA HF is connected, directly after closing VARA HF, and 60s after that.
```console pi@raspberrypi4:/etc/init.d $ netstat | grep tcp && date tcp 0 0 localhost:51220 localhost:36759 ESTABLISHED tcp 0 0 raspberrypi4:47374 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:36759 localhost:51220 ESTABLISHED tcp 0 0 localhost:53971 localhost:8301 ESTABLISHED tcp 0 0 raspberrypi4:45066 ec2-3-208-217-166:https ESTABLISHED tcp 0 0 raspberrypi4:60810 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:56953 localhost:59036 ESTABLISHED tcp 0 0 localhost:59036 localhost:56953 ESTABLISHED tcp 0 0 localhost:8300 localhost:46719 ESTABLISHED tcp 0 0 localhost:8301 localhost:53971 ESTABLISHED tcp 0 0 localhost:46719 localhost:8300 ESTABLISHED Thu 13 Oct 14:34:41 MDT 2022 pi@raspberrypi4:/etc/init.d $ netstat | grep tcp && date tcp 0 0 localhost:51220 localhost:36759 ESTABLISHED tcp 0 0 raspberrypi4:47374 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:36759 localhost:51220 ESTABLISHED tcp 0 0 raspberrypi4:45066 ec2-3-208-217-166:https ESTABLISHED tcp 0 0 raspberrypi4:60810 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:56953 localhost:59036 ESTABLISHED tcp 0 0 localhost:59036 localhost:56953 ESTABLISHED tcp 0 0 localhost:8300 localhost:46719 TIME_WAIT tcp 0 0 localhost:8301 localhost:53971 TIME_WAIT Thu 13 Oct 14:34:47 MDT 2022 ... pi@raspberrypi4:/etc/init.d $ netstat | grep tcp && date tcp 0 0 localhost:51220 localhost:36759 ESTABLISHED tcp 0 0 raspberrypi4:47374 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:36759 localhost:51220 ESTABLISHED tcp 0 0 raspberrypi4:45066 ec2-3-208-217-166:https ESTABLISHED tcp 0 0 raspberrypi4:60810 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:56953 localhost:59036 ESTABLISHED tcp 0 0 localhost:59036 localhost:56953 ESTABLISHED tcp 0 0 localhost:8300 localhost:46719 TIME_WAIT tcp 0 0 localhost:8301 localhost:53971 TIME_WAIT Thu 13 Oct 14:35:44 MDT 2022 pi@raspberrypi4:/etc/init.d $ netstat | grep tcp && date tcp 0 0 localhost:51220 localhost:36759 ESTABLISHED tcp 0 0 raspberrypi4:47374 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:36759 localhost:51220 ESTABLISHED tcp 0 0 raspberrypi4:60810 165.254.191.199:https ESTABLISHED tcp 0 0 localhost:56953 localhost:59036 ESTABLISHED tcp 0 0 localhost:59036 localhost:56953 ESTABLISHED Thu 13 Oct 14:35:48 MDT 2022 pi@raspberrypi4:/etc/init.d $ ```Things we've tried
- The Linux kernel has some network variables that we could (in theory) change to help us fix/diagnose this issue (1). Note: Some are integer vars, some are boolean vars.
- Setting
net.ipv4.tcp_keepalive_time,net.ipv4.tcp_fin_timeout, &sunrpc.tcp_fin_timeoutto int1doesn't seem to change anything(?) with/without network reset (TIME_WAITstill stays for 60s). Doing this also wrecks the internet on the Pi. - Setting
net.ipv4.tcp_tw_reuseto int1(global enable) doesn't change any behavior either. - Some forums suggest forcibly cutting the connection on the TCP port with a program like Killcx. However, this does not address our
TIME_WAITproblem, which is a port "busy" state that arises AFTER the port's connection is cut.
Click to expand: Viewing/editing/updating(?) Linux kernel variables in the terminal.
```bash # view kernel variables sysctl -A -r tcp # show tcp variables sysctl -A -r tw # show some time_wait variables # view current state of individual variables sysctl net.ipv4.tcp_keepalive_time sysctl net.ipv4.tcp_fin_timeout sysctl net.ipv4.tcp_tw_reuse #sysctl net.ipv4.tcp_tw_recycle #not available in RPiOS I think # edit individual variables - these changes do not persist after reboot sudo sysctl -w net.ipv4.tcp_keepalive_time=1 sudo sysctl -w net.ipv4.tcp_fin_timeout=1 sudo sysctl -w sunrpc.tcp_fin_timeout=1 sudo sysctl -w net.ipv4.tcp_tw_reuse=1 # can also view/edit state of individual variables like this sudo nano /proc/sys/net/ipv4/tcp_fin_timeout # reload network (needed for new variables to take effect?) sudo service networking restart #could also run: /etc/init.d/networking restart #reload variables (from config files?) sudo sysctl --system ```Click to expand: Killcx RPiOS installation (fyi. doesn't help)
```console #Killcx only detects VARA connection while it's active, does not disable TIME_WAIT state on ports cd ~/Downloads wget https://cfhcable.dl.sourceforge.net/project/killcx/killcx/1.0.3/killcx-1.0.3.tgz 7z x killcx-1.0.3.tgz sudo apt-get install libnet-rawip-perl libnet-pcap-perl libnetpacket-perl sudo chmod +x killcx-1.0.3/killcx cat /etc/hosts # to confirm that localhost is 127.0.0.1 sudo killcx-1.0.3/./killcx 127.0.0.1:8300 tcp ```Possible solutions
- Kindly ask VARA's dev, EA5HVK, if he would be able to make VARA's TCP/ports/sockets connection routine ignore a
TIME_WAITstate on a port and connect anyway _(similar to the C function "SO_REUSEADDR") (1,2)_ - Create some sort of wrapper that runs instead of VARA which includes an "SOREUSEADDR"-type function, configures VARA's port to be different for each run, and passes traffic to/from VARA-apps? (This would take a lot of work and might be buggy. I wouldn't even know where to begin to make something like this although I think it's theoretically possible)._
- Make a script that monitors port states on Linux and warns users that VARA cannot be run during a 60s countdown window if ports 8300/8301 are found to be in the
TIME_WAITstate. (KM4ACK's idea - he also has a prototype script written to do this). - Make a daemon script that monitors for VARA in the background at all times: When VARA is run, log PID and wait for VARA to close. Upon VARA closing, reset the network with
sudo service networking restart. (This is not a favorable option since it could cause users to lose internet connection / data unexpectedly).
SpudGunMan
pe1rrr
georges
This problem affects Linux/Wine, but does not occur on Windows. Fixing this for Linux would make VARA much more usable for users who do not have Windows.