Open jludwig opened 1 month ago
SOS Report: sos_2024-07-09_23-37-58.tar.gz
Have you checked for firewall settings on both nodes? Looks like the k8s node is struggling to receive anything from the proxmox node.
jludwig@k3s-int-stage-work03:~$ sudo iptables -L -v -n | grep -E "7000|3366|3376"
# Warning: iptables-legacy tables present, use iptables-legacy to see them
745 42076 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3366
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:3376
268 14800 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 7000:7010
547 51601 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:3366
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:3376
214 11560 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport sports 7000:7010
For Proxmox, I actually don't use iptables at all and just rely on OPNsense to do it's job.
On OPNsense, I can look at my live firewall rules, and I see the packets being allowed back and fourth to each node.
Are there any other rules I should look for/add?
Note, after updating the log level to "TRACE", I see this:
2024_07_10 16:11:20.777 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change connection name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 connection:Connecting 2024_07_10 16:11:21.287 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change path name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 local:ipv4:192.168.42.158:7000 peer:ipv4:192.168.42.25:7000 established:yes 2024_07_10 16:11:23.315 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change connection name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 connection:BrokenPipe 2024_07_10 16:11:23.348 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change path name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 local:ipv4:192.168.42.158:7000 peer:ipv4:192.168.42.25:7000 established:no 2024_07_10 16:11:23.353 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change connection name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 connection:Unconnected 2024_07_10 16:11:24.363 [DrbdEventService] TRACE LINSTOR/Satellite - SYSTEM - DRBD 'events2': change connection name:pvc-ac80db5a-c34b-4fe1-b8ec-333dbbf13989 peer-node-id:0 conn-name:pve-epyc-01 connection:Connecting
Edit: I should mention that seeing the packets in OPNsense only happened when I was connecting across vlans.
I did iperf3 tests from both nodes when on the same vlan on port 7000 and I'm getting >10 gbits per second and jitter below .02ms. No lost packets.
I think this may be a drbd issue and I should probably post this there instead.
I did iperf3 tests from both nodes when on the same vlan on port 7000 and I'm getting >10 gbits per second and jitter below .02ms. No lost packets.
Just to make extra sure, did you test it in both directions? I.e. running the iperf3 "server" once on a k8s host and once on a proxmox host?
What is interesting is that from the logs it looks like the proxmox nodes does not see any connection attempts at all, while the k8s worker seems to "talk" to something which does not look like DRBD, hence it timing out eventually.
Other than that, yeah, might be better to open an issue on DRBD. You may want to upgrade to DRBD 9.2.10 on the k8s nodes first just to make sure, even if there does not seem to be any relevant change in the latest release.
Environment
Setup Description
Description
When creating a resource using a PVC that uses the configured storageclass, the DRBD connection between the Kubernetes worker node and the Proxmox node (acting as the external LINSTOR controller) fails to establish. The connection attempts are continuous but unsuccessful.
Observed Behavior
DRBD Connection Failures:
DRBD Status:
Kubernetes Worker Node:
Proxmox Node:
Note: On both nodes, the connections cycle between Unconnected and Connecting states.
Additional data:
Configuration
StorageClass:
LinstorCluster:
LinstorSatelliteConfiguration:
DRBD Resource Configuration (k3s-int-stage-work03):
DRBD Resource Configuration (pve-epyc-01):
Additional Information
Troubleshooting Steps Taken
Questions