Connman is unfit for use with static network interface configurations

insanemal commented 3 years ago

Version (k3OS / kernel) v1.19.2+k3s1 1.4.0-k3s1

Architecture x86_64 (but probably any)

Describe the bug If you configure connman to have multiple static interfaces and no dynamic interfaces it configures the default route using eth1 regardless of any of the other config provided. Not only that connman does not have a facility to tell it what to use as a default route. It appears to be designed with either DHCP or single static interface use in mind. Oh also if you try and change it using ip route or the "deprecated" route command, connman conveniently keeps undoing all attempts to fix the routing for about the first 2-3 mins of node boot.

To Reproduce Using a script that runs at boot configure at least eth0 and eth1 setting the IP, Gateway and mask on both (or more)

Expected behavior Have an option to specify a default route and set that as the default. Failing that I'd kinda like it to set things once and stop undoing attempts to fix the routing.

Actual behavior It always and without failure sets whatever you specified as the gateway for eth1 as the default gateway. Regardless of the number of interfaces. Attempts to set the routing via script or manually by logging in after boot are undone by connman until some unspecified time out. Then you can change things and they stick.

Additional context Visualised, physical it doesn't seem to matter. It just seems that connman was not designed for anything but the most simple of network configurations. It's not really fit for purpose. I mean come on, this was a feature of the old if-up bash scripts. If it can't do simple things like set a specific default gateway, what good is it?

himslm01 commented 3 years ago

Further to the above, connman cannot set static routes. This further highlights that it is not fit to be used in a server environment.

nsmith5 commented 3 years ago

Some helpful references I landed on thinking about this issue:

Was hoping there was some sneaky setting for static routes or configuration but I couldn't find any..

insanemal commented 3 years ago

This is causing me some deal of issue. I have had nodes randomly reset their routing during normal operation. This renders them uncontactable.

So this needs to be reviewed

pr0ton11 commented 3 years ago

We need a better solution than connman. I have the same issue. I want to connect all k3s nodes over a private connection and only have ingress use the public ip adress.

Everytime I configure connman with the static ip adress on eth1, I can not connect ssh anymore, because default route goes over the interface without any static route. And because the node is physical I can't even change them. This is unusable.

neatherweb commented 3 years ago

I agree this seems a bit too much of an opinionated network manager. In the interim it may be suitable and worth keeping for IOT or micro devices but with a option to disable it and use something better or even if-up/down scripts from cloud-int userdata. At the moment, I have found the default behaviour with using static config (via write_file to connman service) results in DHCP and static IPs (as secondary) on a interface and also it messes with the hostname. For me I found I have to use run_cmd as a work around just for static IP to be functional. disable/enable of the ethernet is needed to clear the DHCP IP address but after the disable, hostname reverts to the autogenerated name - so you also need to manually fix this. (last tested with v0.21.0-k3s1r0)

write_files:
          - content: |
              [service_eth0]
              Type = ethernet
              IPv4 = 192.168.1.107/255.255.255.0/192.168.1.1
              Nameservers = 192.168.1.106
              SearchDomains = internal.mydomain.com
              Timeservers = 0.au.pool.ntp.org,1.au.pool.ntp.org,2.au.pool.ntp.org,3.au.pool.ntp.org
              Domain = internal.mydomain.com
            path: /var/lib/connman/eth0.config
        run_cmd:
          - connmanctl disable ethernet
          - hostname k3os1
          - connmanctl enable ethernet

see also #560

cliv commented 2 years ago

From connman's manpages - It also looks like there's no way to set MTU on interfaces (eg for jumbo frames support).

insanemal commented 2 years ago

Why is this ticket being ignored? Is it safe to assume that k3os, like RancherOS before it is about to be dumped to no fanfare? You all seem very good at making new "products" only to unoffically abandon them years out from eventually replacing them with another new hotness that you then also go on to ignore, half completed

bitmage commented 2 years ago

I'm glad @insanemal is mad so I don't have to be.

insanemal commented 2 years ago

Honestly it's an absolute joke at this point. Looks like RKE2 is the new hotness.

Personally I think I'm going to stop using Rancher's "products" outside of the manager Web gui thing as they are worse than Google when it comes to abandoning projects.

Seriously how is this ticket still open? It's over a year old at this point. Tell me your team has no direction without telling me your team has no direction

rancher / k3os

Connman is unfit for use with static network interface configurations #629