Full CNI DHCP plugin support

[dtrudg]

Version of Singularity

main / 4.1

Describe the bug

If a CNI plugin is run using dhcp IPAM, two portions of the dhcp plugin are in play:

• The dhcp client plugin is used to setup the container network namespace.
• The dhcp plugin daemon is run on the host, and will hold / renew reservations for addresses consumed by containers.

The dhcp client plugin will communicate with the daemon, and pass the path to a bound network namespace associated with the container. The daemon requires access to this network namespace.

When setting up CNI networks we currently:

• Add a bind point from /proc/self/ns/net into the singularity session directory.
• Call the CNI network setup using the path in the session directory.

There are two problems here:

• The bind is not actually applied at CNI setup time, leading to an error such as:

  FATAL:   container creation failed: plugin type="macvlan" failed (add): error calling DHCP.Allocate: failed to Statfs "/usr/local/var/singularity/mnt/session/netns": no such file or directory

• The session directory is not accessible to the dhcp daemon on the host, so it cannot access the container's network namespace.

To support the dhcp plugin we must:

• Bind mount the network namespace to a location that is shared with / accessible from the host - not only inside the per-container session mount namespace.
• Ensure the bind happens before the dhcp plugin is called, and that the correct path is passed through to the plugin, which will in turn pass it to the dhcp daemon on the host.

[dtrudg]

A full fix for this is complex, requiring invasive changes to the runtime.

As an initial workaround, will add the ability for admins to specify existing network namespaces which a user may join, and a flag for a user to request namespace join rather than creation.

[dtrudg]

An alternative workaround may be to pass /proc/<pid>/ns paths to the CNI code that eventually passes them to the DHCP daemon... in the situation that it can be guaranteed that the DHCP CNI daemon and the container are in the same PID namespace.

We have the ability to disable PID namespace creation in singularity.conf, in which case only a user with privilege can run a container in a different PID namespace. This still needs careful thought for security concerns.. i.e. is PID reuse a possible issue here?

[dtrudg]

For CE 4.2 we're going for the workaround in #3189 - so a netns with CNI DHCP can be setup separately, and then joined.