Closed nberlee closed 1 year ago
You probably want to know this @twelho ?
Do not worry about me, I already worked around it (set machine>network>hostname)
confirmed, same thing in my lab
Talos is supposed to send another INFORM
packet now... I agree it's a regression we need to fix, just trying to gather more info.
Can you please provide output of talosctl logs controller-runtime | grep dhcp
?
I undid my workaround on one node, rebooted and then did
talosctl logs controller-runtime -n 192.168.147.5| grep dhcp
192.168.147.5: 2023-04-03T10:18:13.108Z DEBUG starting operator {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4/eth0"}
192.168.147.5: 2023-04-03T10:18:13.109Z DEBUG DHCP REQUEST {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
192.168.147.5: 2023-04-03T10:18:13.110Z DEBUG DHCP ACK {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0", "dhcp": "opcode: BootReply, hwtype: Ethernet, hopcount: 0, transaction ID: 0x5b947287, num seconds: 0, flags: Unicast (0x00), client IP: 0.0.0.0, your IP: 192.168.147.5, server IP: 192.168.147.30, gateway IP: 0.0.0.0, client MAC: 52:54:00:<SANITIZED>, server hostname:, bootfile name:, options:, Subnet Mask: ffffffe0, Router: 192.168.147.30, Domain Name Server: 9.9.9.9, NTP Servers: 192.168.147.30, IP Addresses Lease Time: 336h0m0s, DHCP Message Type: ACK, Server Identifier: 192.168.147.30"}
192.168.147.5: 2023-04-03T10:18:13.565Z DEBUG DHCP INFORM {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
So looks like there's no response to INFORM
whatsoever... I'll try to see what I can do
Wait, that may not be it... i changed my INPUT firewall on the mikrotik... I created an extra rule to allow non broadcast traffic back....
192.168.147.5: 2023-04-03T10:29:41.152Z DEBUG starting operator {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4/eth0"}
192.168.147.5: 2023-04-03T10:29:41.154Z DEBUG DHCP REQUEST {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
192.168.147.5: 2023-04-03T10:29:41.155Z DEBUG DHCP ACK {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0", "dhcp": "opcode: BootReply, hwtype: Ethernet, hopcount: 0, transaction ID: 0x58c96f8f, num seconds: 0, flags: Unicast (0x00), client IP: 0.0.0.0, your IP: 192.168.147.5, server IP: 192.168.147.30, gateway IP: 0.0.0.0, client MAC: 52:54:00:a0:6e:55, server hostname:, bootfile name:, options:, Subnet Mask: ffffffe0, Router: 192.168.147.30, Domain Name Server: 9.9.9.9, NTP Servers: 192.168.147.30, IP Addresses Lease Time: 336h0m0s, DHCP Message Type: ACK, Server Identifier: 192.168.147.30"}
192.168.147.5: 2023-04-03T10:29:41.649Z DEBUG DHCP INFORM {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
192.168.147.5: 2023-04-03T10:30:16.652Z WARN hostname request failed {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "error": "got an error while processing the request: no matching response packet received", "link": "eth0"}
192.168.147.5: 2023-04-03T10:30:17.656Z DEBUG detected hostname change {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "old": "", "new": "talos-192-168-147-5"}
192.168.147.5: 2023-04-03T10:30:17.656Z DEBUG restarting DHCP sequence due to hostname change {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "dhcp_hostname": []}
192.168.147.5: 2023-04-03T10:30:17.658Z DEBUG DHCP REQUEST {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
192.168.147.5: 2023-04-03T10:30:17.659Z DEBUG DHCP ACK {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0", "dhcp": "opcode: BootReply, hwtype: Ethernet, hopcount: 0, transaction ID: 0x5e53deab, num seconds: 0, flags: Unicast (0x00), client IP: 0.0.0.0, your IP: 192.168.147.5, server IP: 192.168.147.30, gateway IP: 0.0.0.0, client MAC: 52:54:00:a0:6e:55, server hostname:, bootfile name:, options:, Subnet Mask: ffffffe0, Router: 192.168.147.30, Domain Name Server: 9.9.9.9, NTP Servers: 192.168.147.30, IP Addresses Lease Time: 336h0m0s, DHCP Message Type: ACK, Server Identifier: 192.168.147.30"}
192.168.147.5: 2023-04-03T10:30:17.686Z DEBUG DHCP INFORM {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
But Same outcome...
DHCP renewals should be non-broadcast by the way, but in any case this feels like INFORM
gets dropped. I will see what I can do to drop INFORM
packets and unify both flows.
Ok looks like, with inform it also does not work... but this is also clearly mikrotik at fault
DHCP Inform packets where received after I removed DHCP Snooping from the bridge (seems a like a filter which only looks at SRC and DST ports)
192.168.147.5: 2023-04-03T12:23:22.565Z DEBUG DHCP INFORM {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"}
192.168.147.5: 2023-04-03T12:23:22.566Z DEBUG DHCP ACK {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0", "dhcp": "opcode: BootReply, hwtype: Ethernet, hopcount: 0, transaction ID: 0x018918ac, num seconds: 0, flags: Unicast (0x00), client IP: 192.168.147.5, your IP: 0.0.0.0, server IP: 192.168.147.30, gateway IP: 0.0.0.0, client MAC: 52:54:00:<SANITIZED>, server hostname:, bootfile name:, options:, DHCP Message Type: ACK, Server Identifier: 192.168.147.30"}
14:23:22 dhcp,debug dhcp-Servers sending ack with id 3876318372 to 192.168.147.5
14:23:22 dhcp,debug,packet ciaddr = 0.0.0.0
14:23:22 dhcp,debug,packet yiaddr = 192.168.147.5
14:23:22 dhcp,debug,packet siaddr = 192.168.147.30
14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED>
14:23:22 dhcp,debug,packet Subnet-Mask = 255.255.255.224
14:23:22 dhcp,debug,packet Router = 192.168.147.30
14:23:22 dhcp,debug,packet Domain-Server = 9.9.9.9
14:23:22 dhcp,debug,packet NTP-Server = 192.168.147.30
14:23:22 dhcp,debug,packet Address-Time = 1209600
14:23:22 dhcp,debug,packet Msg-Type = ack
14:23:22 dhcp,debug,packet Server-Id = 192.168.147.30
14:23:22 dhcp,debug dhcp-Servers received inform id 2887289089 from 192.168.147.5 ''
14:23:22 dhcp,debug,packet ciaddr = 192.168.147.5
14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED>
14:23:22 dhcp,debug,packet Msg-Type = inform
14:23:22 dhcp,debug,packet Parameter-List = Host-Name,Domain-Name
14:23:22 dhcp,debug dhcp-Servers sending ack with id 2887289089 to 192.168.147.5
14:23:22 dhcp,debug,packet ciaddr = 192.168.147.5
14:23:22 dhcp,debug,packet siaddr = 192.168.147.30
14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED>
14:23:22 dhcp,debug,packet Msg-Type = ack
14:23:22 dhcp,debug,packet Server-Id = 192.168.147.30
In the inform I see Host-Name, but its not sent...
I'm testing a fix right now, removing INFORM
Ok looks like, with inform it also does not work... but this is also clearly mikrotik at fault
DHCP Inform packets where received after I removed DHCP Snooping from the bridge (seems a like a filter which only looks at SRC and DST ports)
192.168.147.5: 2023-04-03T12:23:22.565Z DEBUG DHCP INFORM {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0"} 192.168.147.5: 2023-04-03T12:23:22.566Z DEBUG DHCP ACK {"component": "controller-runtime", "controller": "network.OperatorSpecController", "operator": "dhcp4", "link": "eth0", "dhcp": "opcode: BootReply, hwtype: Ethernet, hopcount: 0, transaction ID: 0x018918ac, num seconds: 0, flags: Unicast (0x00), client IP: 192.168.147.5, your IP: 0.0.0.0, server IP: 192.168.147.30, gateway IP: 0.0.0.0, client MAC: 52:54:00:<SANITIZED>, server hostname:, bootfile name:, options:, DHCP Message Type: ACK, Server Identifier: 192.168.147.30"}
14:23:22 dhcp,debug dhcp-Servers sending ack with id 3876318372 to 192.168.147.5 14:23:22 dhcp,debug,packet ciaddr = 0.0.0.0 14:23:22 dhcp,debug,packet yiaddr = 192.168.147.5 14:23:22 dhcp,debug,packet siaddr = 192.168.147.30 14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED> 14:23:22 dhcp,debug,packet Subnet-Mask = 255.255.255.224 14:23:22 dhcp,debug,packet Router = 192.168.147.30 14:23:22 dhcp,debug,packet Domain-Server = 9.9.9.9 14:23:22 dhcp,debug,packet NTP-Server = 192.168.147.30 14:23:22 dhcp,debug,packet Address-Time = 1209600 14:23:22 dhcp,debug,packet Msg-Type = ack 14:23:22 dhcp,debug,packet Server-Id = 192.168.147.30 14:23:22 dhcp,debug dhcp-Servers received inform id 2887289089 from 192.168.147.5 '' 14:23:22 dhcp,debug,packet ciaddr = 192.168.147.5 14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED> 14:23:22 dhcp,debug,packet Msg-Type = inform 14:23:22 dhcp,debug,packet Parameter-List = Host-Name,Domain-Name 14:23:22 dhcp,debug dhcp-Servers sending ack with id 2887289089 to 192.168.147.5 14:23:22 dhcp,debug,packet ciaddr = 192.168.147.5 14:23:22 dhcp,debug,packet siaddr = 192.168.147.30 14:23:22 dhcp,debug,packet chaddr = 52:54:00:<SANITIZED> 14:23:22 dhcp,debug,packet Msg-Type = ack 14:23:22 dhcp,debug,packet Server-Id = 192.168.147.30
In the inform I see Host-Name, but its not sent...
This indeed looks like Mikrotik's implementation is at fault, everything looks as it should from Talos' side. As mentioned in https://github.com/siderolabs/talos/pull/5897 I could only test this against dnsmasq
, which works correctly in this case. These implementation issues are really unfortunate...
@smira Thank You!
I've created a bug report at support@mikrotik.com: MikroTik support #[SUP-112536]: DHCP Inform Ack does not include Host-Name
Bug Report
In the DHCP request from v1.4.0-alpha.4 the parameter list no longer asks for a Host-Name. If a node with a DHCP reservation with option 12 is updated, the hostname is changed to the talos-ip-with-dashes format. (when machine-network-hostname is not set)
Description
After the upgrade (from 1.3.2) the worker node which used to be
talos-home-5
is nowtalos-192-168-147-5
in kubernetes. It used to get the hostname from DHCP but now it does not.I also upgraded a controlplane which is now called:
talos-192-168-147-13
instead oftalos-home-3
I am use a DHCP server from the Mikrotik router. Which is capable to send Option 12 (hostname) to the client, this way, I centralize my host-naming with reservation. This used to work in Talos 1.3 and before.
Logs
Mikrotik DHCP server log in debug mode of 1.4.0-alpha.4 node reboot: Please note the Parameter-List does not contain Host-Name in
alpha.4
DHCP debug mode of 1.3.2 link down up:
Environment