search

clincha-org / clincha

Configuration and monitoring of clinch-home infrastructure
https://clinch-home.com
1 stars 1 forks source link

Factorio server crashes after a short time #115

Open clincha opened 10 months ago

clincha commented 10 months ago

After deploying Factorio to the cluster and logging in, I can play for a few minutes and then I am disconnected from the game.

clincha commented 10 months ago

I have forked the factoriotools/factorio-docker repository and enabled verbose logging so that I can see if that has any errors in it. clincha/factorio-docker contains my code changes

clincha commented 10 months ago

The verbose output generates a lot of logs so I've put the full output on Pastebin.

The important line is here

78884.750 Info UnparsedNetworkMessage.cpp:91: Invalid packet (type ClientToServerHeartbeat) received from IP ADDR:({10.1.2.103:40054}): unknown sender

clincha commented 10 months ago

iPerf is showing some pretty consistent retries happening on the interface. These could be related.

clincha@angus-main:~$ iperf3 -c clinch-home.com -t 120
Connecting to host clinch-home.com, port 5201
[  5] local 192.168.1.225 port 56708 connected to 185.23.254.226 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  2.20 MBytes  18.5 Mbits/sec    0    118 KBytes       
[  5]   1.00-2.00   sec  2.53 MBytes  21.3 Mbits/sec    0    231 KBytes       
[  5]   2.00-3.00   sec  2.84 MBytes  23.8 Mbits/sec    0    343 KBytes       
[  5]   3.00-4.00   sec  2.41 MBytes  20.2 Mbits/sec    0    456 KBytes       
[  5]   4.00-5.00   sec  2.97 MBytes  24.9 Mbits/sec    0    570 KBytes       
[  5]   5.00-6.00   sec  2.41 MBytes  20.2 Mbits/sec    0    682 KBytes       
[  5]   6.00-7.00   sec  2.50 MBytes  21.0 Mbits/sec    0    795 KBytes       
[  5]   7.00-8.00   sec  2.50 MBytes  21.0 Mbits/sec    0    908 KBytes       
[  5]   8.00-9.00   sec  2.50 MBytes  21.0 Mbits/sec    0   1022 KBytes       
[  5]   9.00-10.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.11 MBytes       
[  5]  10.00-11.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.21 MBytes       
[  5]  11.00-12.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.31 MBytes       
[  5]  12.00-13.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.42 MBytes       
[  5]  13.00-14.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.53 MBytes       
[  5]  14.00-15.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.68 MBytes       
[  5]  15.00-16.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.97 MBytes       
[  5]  16.00-17.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.33 MBytes       
[  5]  17.00-18.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.73 MBytes       
[  5]  18.00-19.00  sec  2.50 MBytes  21.0 Mbits/sec    0   3.18 MBytes       
[  5]  19.00-20.00  sec  1.25 MBytes  10.5 Mbits/sec  151   2.43 MBytes       
[  5]  20.00-21.00  sec  3.75 MBytes  31.5 Mbits/sec   48   2.43 MBytes       
[  5]  21.00-22.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.50 MBytes       
[  5]  22.00-23.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.67 MBytes       
[  5]  23.00-24.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.83 MBytes       
[  5]  24.00-25.00  sec  1.25 MBytes  10.5 Mbits/sec   61   2.67 MBytes       
[  5]  25.00-26.00  sec  0.00 Bytes  0.00 bits/sec  123   2.05 MBytes       
[  5]  26.00-27.00  sec  5.00 MBytes  42.0 Mbits/sec    0   2.09 MBytes       
[  5]  27.00-28.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.18 MBytes       
[  5]  28.00-29.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.27 MBytes       
[  5]  29.00-30.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.34 MBytes       
[  5]  30.00-31.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.40 MBytes       
[  5]  31.00-32.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.44 MBytes       
[  5]  32.00-33.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.47 MBytes       
[  5]  33.00-34.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.48 MBytes       
[  5]  34.00-35.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.49 MBytes       
[  5]  35.00-36.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.49 MBytes       
[  5]  36.00-37.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.49 MBytes       
[  5]  37.00-38.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.50 MBytes       
[  5]  38.00-39.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.51 MBytes       
[  5]  39.00-40.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.52 MBytes       
[  5]  40.00-41.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.55 MBytes       
[  5]  41.00-42.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.59 MBytes       
[  5]  42.00-43.00  sec  1.25 MBytes  10.5 Mbits/sec   19   2.29 MBytes       
[  5]  43.00-44.00  sec  3.75 MBytes  31.5 Mbits/sec   30   1.84 MBytes       
[  5]  44.00-45.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.94 MBytes       
[  5]  45.00-46.00  sec  1.25 MBytes  10.5 Mbits/sec   52   1.80 MBytes       
[  5]  46.00-47.00  sec  3.75 MBytes  31.5 Mbits/sec   20   1.44 MBytes       
[  5]  47.00-48.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.50 MBytes       
[  5]  48.00-49.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.58 MBytes       
[  5]  49.00-50.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.64 MBytes       
[  5]  50.00-51.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.68 MBytes       
[  5]  51.00-52.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.71 MBytes       
[  5]  52.00-53.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.73 MBytes       
[  5]  53.00-54.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.73 MBytes       
[  5]  54.00-55.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.74 MBytes       
[  5]  55.00-56.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.74 MBytes       
[  5]  56.00-57.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.74 MBytes       
[  5]  57.00-58.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.75 MBytes       
[  5]  58.00-59.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.76 MBytes       
[  5]  59.00-60.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.79 MBytes       
[  5]  60.00-61.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.82 MBytes       
[  5]  61.00-62.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.88 MBytes       
[  5]  62.00-63.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.95 MBytes       
[  5]  63.00-64.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.05 MBytes       
[  5]  64.00-65.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.17 MBytes       
[  5]  65.00-66.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.32 MBytes       
[  5]  66.00-67.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.50 MBytes       
[  5]  67.00-68.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.71 MBytes       
[  5]  68.00-69.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.95 MBytes       
[  5]  69.00-70.00  sec  1.25 MBytes  10.5 Mbits/sec   86   2.15 MBytes       
[  5]  70.00-71.00  sec  3.75 MBytes  31.5 Mbits/sec   82   2.15 MBytes       
[  5]  71.00-72.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.27 MBytes       
[  5]  72.00-73.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.43 MBytes       
[  5]  73.00-74.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.57 MBytes       
[  5]  74.00-75.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.70 MBytes       
[  5]  75.00-76.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.80 MBytes       
[  5]  76.00-77.00  sec  1.25 MBytes  10.5 Mbits/sec   16   2.64 MBytes       
[  5]  77.00-78.00  sec  0.00 Bytes  0.00 bits/sec   88   2.00 MBytes       
[  5]  78.00-79.00  sec  5.00 MBytes  42.0 Mbits/sec    0   2.01 MBytes       
[  5]  79.00-80.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.12 MBytes       
[  5]  80.00-81.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.21 MBytes       
[  5]  81.00-82.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.29 MBytes       
[  5]  82.00-83.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.34 MBytes       
[  5]  83.00-84.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.38 MBytes       
[  5]  84.00-85.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.41 MBytes       
[  5]  85.00-86.00  sec  1.25 MBytes  10.5 Mbits/sec   14   2.25 MBytes       
[  5]  86.00-87.00  sec  1.25 MBytes  10.5 Mbits/sec    5   1.69 MBytes       
[  5]  87.00-88.00  sec  3.75 MBytes  31.4 Mbits/sec    0   1.74 MBytes       
[  5]  88.00-89.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.82 MBytes       
[  5]  89.00-90.00  sec  1.25 MBytes  10.5 Mbits/sec    0   1.90 MBytes       
[  5]  90.00-91.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.95 MBytes       
[  5]  91.00-92.00  sec  2.50 MBytes  21.0 Mbits/sec    0   1.99 MBytes       
[  5]  92.00-93.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.03 MBytes       
[  5]  93.00-94.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.04 MBytes       
[  5]  94.00-95.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.05 MBytes       
[  5]  95.00-96.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.05 MBytes       
[  5]  96.00-97.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.06 MBytes       
[  5]  97.00-98.00  sec  1.25 MBytes  10.5 Mbits/sec    0   2.06 MBytes       
[  5]  98.00-99.00  sec  2.50 MBytes  21.0 Mbits/sec    0   2.07 MBytes       
[  5]  99.00-100.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.08 MBytes       
[  5] 100.00-101.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.11 MBytes       
[  5] 101.00-102.00 sec  1.25 MBytes  10.5 Mbits/sec    0   2.14 MBytes       
[  5] 102.00-103.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.20 MBytes       
[  5] 103.00-104.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.28 MBytes       
[  5] 104.00-105.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.37 MBytes       
[  5] 105.00-106.00 sec  1.25 MBytes  10.5 Mbits/sec    0   2.50 MBytes       
[  5] 106.00-107.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.65 MBytes       
[  5] 107.00-108.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.83 MBytes       
[  5] 108.00-109.00 sec  2.50 MBytes  21.0 Mbits/sec    0   3.04 MBytes       
[  5] 109.00-110.00 sec  0.00 Bytes  0.00 bits/sec  162   2.14 MBytes       
[  5] 110.00-111.00 sec  3.75 MBytes  31.4 Mbits/sec   68   2.16 MBytes       
[  5] 111.00-112.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.29 MBytes       
[  5] 112.00-113.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.44 MBytes       
[  5] 113.00-114.00 sec  1.25 MBytes  10.5 Mbits/sec    0   2.59 MBytes       
[  5] 114.00-115.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.71 MBytes       
[  5] 115.00-116.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.81 MBytes       
[  5] 116.00-117.00 sec  1.25 MBytes  10.5 Mbits/sec   14   2.58 MBytes       
[  5] 117.00-118.00 sec  3.75 MBytes  31.5 Mbits/sec   78   2.00 MBytes       
[  5] 118.00-119.00 sec  1.25 MBytes  10.5 Mbits/sec    0   2.05 MBytes       
[  5] 119.00-120.00 sec  2.50 MBytes  21.0 Mbits/sec    0   2.14 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-120.00 sec   264 MBytes  18.5 Mbits/sec  1117             sender
[  5]   0.00-120.97 sec   263 MBytes  18.3 Mbits/sec                  receiver
clincha commented 10 months ago

Timing doesn't seem to be related

clincha commented 10 months ago

I tried on another interface and I'm getting the same issue so I don't think that it's the NICs

clincha commented 10 months ago

I tried disabling the "suspicious activity" blocker on the firewall but that didn't change anything

clincha commented 9 months ago

I'm installing the ksniff plugin to inspect the traffic.

I run this command on the master node

k sniff factorio-756647974d-n4h6f -p -n default -o output.pcap

Then join the multiplayer server from my client and wait until the error occurs.

Start capture: 16:09:35 Joined: 16:09:42 Error: 16:10:08 Error end: 16:10:29 End capture: 16:10:38

clincha commented 9 months ago

Made a post on the forums: https://forums.factorio.com/viewtopic.php?f=49&t=108772