Open ksanfs opened 2 months ago
We previously encountered this issue about a year ago, which resulted in a significant outage. As a consequence, we decided not to use Talos. Unfortunately, when we reached out to Hetzner support, they informed us that since Talos is not one of their officially supported operating systems, they were unable to assist us with the problem.
It sounds weird, but if the linkState:
is down
as it is in the output you provided, it is equivalent to the cable being unplugged (with the physical network card), so Talos stops the DHCP client in this case (as the address is not usable).
The workaround might be to assign the IP statically in the machine config, but I believe even with that if the linkState
is down, the link won't be still used to send the actual packets, so the fact whether the IP is assigned or not doesn't really matter.
We're experiencing this as well. I can reproduce the exact same behavior, workaround and output @ksanfs provided earlier. Is there anything we can provide to help debug?
Bug Report
Description
We have observed an issue with Hetzner Cloud nodes where the private IP address assigned to the eth1 interface is sometimes lost after Hetzner performs maintenance on the nodes. This issue occurs intermittently and affects the stability and connectivity of the affected nodes.
Steps to Reproduce:
Expected Behavior: The eth1 interface should retain its assigned private IP address after Hetzner maintenance, and the link should be in the "up" state.
Actual Behavior: After Hetzner maintenance, the eth1 interface loses its assigned private IP address. The link is shown as "up" in the linkspec but "down" in the output of
talosctl get link
. This results in a loss of connectivity for the affected nodes.talosctl get linkspec -o yaml
:talosctl get link -o yaml
:talosctl dmesg
:Workaround:
Rebooting the affected nodes seems to resolve the issue temporarily, as the private IP address is reassigned to the eth1 interface after the reboot. Manually detaching the private IP on hetzner cloud website then adding it back resolves the issue temporarily too.
Additional Information:
Environment