search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
262 stars 59 forks source link

Automatic Network Configuration in Cloud with NetworkManager (nm-cloud-setup) #320

Open dustymabe opened 4 years ago

dustymabe commented 4 years ago

The networkmanager community reached out to the Fedora Cloud community to ask about a proposed nm-cloud-setup utility/service: https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/message/FSQR6KL4KA37WHTUAXL774SXSWIBSYGI/

dustymabe commented 4 years ago

cc @lucab @ajeddeloh @bgilbert @jlebon

lucab commented 4 years ago

The tool they started writing may have indeed quite some overlap with Afterburn, but at the moment they seem to target a different niche/usecase.

From a quick skim, relevant differences/points from my side are:

As it stands, nm-cloud-setup seems to be designed as a recurrently-running network-configuration reconciliation tool, while Afterburn network logic tries to take care of one-time bootstrapping the network in early-boot.

/cc @thom311 @bengal

thom311 commented 4 years ago

As it stands, nm-cloud-setup seems to be designed as a recurrently-running network-configuration reconciliation tool, while Afterburn network logic tries to take care of one-time bootstrapping the network in early-boot.

nm-cloud-setup can be called once/manually, but I think a main purpose to pick up changes dynamically. For that it can be triggered via a systemd.timer or a NetworkManager dispatcher script. If you can afford to pre-generate static configuration early at boot, then you don't really need this tool. You could instead just pre-generate the static connection profiles.

Another difference is that the configuration created by nm-cloud-setup is not persisted to file system (not even to /run/NetworkManager). It uses the Reapply() API of NetworkManager to perform runtime only changes. That was done intentionally, it seems that the information that we fetch from the meta data should not be persisted to disk. What is does is similar to calling nmcli device "$IFNAME" modify ipv4.routes ... ipv4.addresses ... ipv4.rules ....

I think another purpose of the tool is to provide a solution for users who use NetworkManager but don't use something like cloud-init/Afterburn.

lucab commented 3 years ago

Some time ago we did some work to add support in Ignition for platform-specific fragments base.platform.d/ in (https://github.com/coreos/ignition/pull/1108) and to verify whether it was powerful enough to enable+configure nm-cloud-setup.

The result of that exploration (https://github.com/coreos/fedora-coreos-config/pull/760) was positive, so there should be no further blockers if we want to proceed. In the meanwhile, upstream added a docpage with all the details about that component: https://networkmanager.pages.freedesktop.org/NetworkManager/NetworkManager/nm-cloud-setup.html.

However we didn't reach consensus on whether we wanted to 1) ship the binary, plus 2) ship the fragment for auto-configuration, plus 3) enable the timer unit by default on FCOS. I'm putting this up for discussion for the next IRC meeting.

lucab commented 3 years ago

In the last IRC meeting we reached consensus that the next steps for nm-cloud-setup integration are:

cgwalters commented 3 years ago

xref https://bugzilla.redhat.com/show_bug.cgi?id=1977984#c56 - may have some negative interactions with OpenShift SDN and I suspect more generally anyone doing nontrivial networking setup outside of the OS.

If we did add this it may warrant a "provisioning discontinuity" at least where only new nodes get it? (like cgroups v2)

Of note though if we did add it, disabling it should just be masking the systemd unit.