Cilium (and maybe other CNIs) assume that host networking is setup to route packets properly.
In more creative setups, this is difficult because we have no way to provision more complex stuff on Talos, i.e. routing tables, differently scoped addresses, routing rules
For example my particular setup requires
1) creating a routing table with a name in rt_tables or an arbitrary id
echo 20 extsvc >> /etc/iproute2/rt_tables
2) creating routing table contents
ip route add 10.64.4.0/24 dev eth0.1004 table extsvc
ip route replace default via 10.64.4.1 table extsvc
3) creating a routing rule
ip rule add from 10.64.4.0/24 lookup extsvc prio 20
Without this, we get assymetrical routing of everything going from the VIP (10.64.4.x) via default gw on internal-only VLAN.
Another issue is setting differently scoped addresses or routes on the interface (Cilium needs at least one IP on an interface to consider using it), which by itself is not a Talos problem, but demonstrates the limits of what MachineConfig can do now.
Please provide at least a somewhat clean way to do this - it makes no sense to APIfy everything linux routing can do, but routing tables and IPs would be worthwhile, and providing a way to execute arbitrary iproute2 commands for the rest would cover most use cases.
smira
commented
3 months ago
Feature Request
Description
Cilium (and maybe other CNIs) assume that host networking is setup to route packets properly. In more creative setups, this is difficult because we have no way to provision more complex stuff on Talos, i.e. routing tables, differently scoped addresses, routing rules
For example my particular setup requires 1) creating a routing table with a name in rt_tables or an arbitrary id echo 20 extsvc >> /etc/iproute2/rt_tables 2) creating routing table contents ip route add 10.64.4.0/24 dev eth0.1004 table extsvc ip route replace default via 10.64.4.1 table extsvc 3) creating a routing rule ip rule add from 10.64.4.0/24 lookup extsvc prio 20
Without this, we get assymetrical routing of everything going from the VIP (10.64.4.x) via default gw on internal-only VLAN.
Another issue is setting differently scoped addresses or routes on the interface (Cilium needs at least one IP on an interface to consider using it), which by itself is not a Talos problem, but demonstrates the limits of what MachineConfig can do now.
Please provide at least a somewhat clean way to do this - it makes no sense to APIfy everything linux routing can do, but routing tables and IPs would be worthwhile, and providing a way to execute arbitrary iproute2 commands for the rest would cover most use cases.
Thank you.