search

lxc / incus

Powerful system container and virtual machine manager
https://linuxcontainers.org/incus
Apache License 2.0
2.43k stars 192 forks source link

Containers that use systemd-networkd use connectivity after a few days #1046

Closed cquike closed 1 month ago

cquike commented 1 month ago

Required information

Issue description

We have several long running containers and we observe that almost once a day one or more machines lose their IPv4 address. A workaround is either to stop and start the containers again or restart the systemd-networkd service inside the container. This problems seems to happen with Ubuntu 24.04 and Fedora 40 which both use systemd-networkd as I can tell. We also have Debian 11 and 12 machines running in the same incus cluster which seem to be unaffected.

We cannot see anything in the logs that helps us to troubleshoot the issue. The only thing I see in the container log is this:

Jul 25 06:05:13 hostname systemd-networkd-wait-online[598082]: Timeout occurred while waiting for network connectivity.

The logs of the containers as shown by incus info --show-logare empty.

The cluster has 4 members and the problem happens in any of the incus hosts. The network configuration for all hosts is a bridge (incusbr0) which use all the containers.

Steps to reproduce

  1. Create a Fedora 40 or Ubuntu 24.04 container in a cluster.
  2. Wait a couple of days until the IPv4 address of the container disappears as shown by incus ls

Information to attach

cquike commented 1 month ago

incus_info.txt Output of incus info attached.

stgraber commented 1 month ago

I've had production clusters running Ubuntu 24.04 containers using networkd with some instances having been up for 2-3 months and never saw any IP disappear, so there's more to this than just some issue with those images.

How many containers are running on those systems?

cquike commented 1 month ago

Around 20 containers are run in a 4 nodes cluster, so on average 4-5 containers per host.

cquike commented 1 month ago

These hosts are also running (non-clustered) LXD daemons for some legacy containers. I hope that it does not cause interference between them.

stgraber commented 1 month ago

Can you show iptables -L -n -v on one such system? It's not impossible that things temporary break the INPUT table.

On very busy systems, I've also seen containers that share uid/gid map run into some issues with netlink communications which can lead to what you're seeing. So you may want to try using security.idmap.isolated=true to get your containers to each have their own uid/gid range.

cquike commented 1 month ago

iptables does not seem to show anything interesting:

# iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
cquike commented 1 month ago

Regarding the idmap: I do have the following configuration in all the containers:

config:
  raw.idmap: |-
    uid 1000-10000 1000-10000
    gid 1000-10000 1000-10000

in order to map the container users to the host ones. Is the security.idmap.isolated=true configuration option compatible with that?

stgraber commented 1 month ago

in order to map the container users to the host ones. Is the security.idmap.isolated=true configuration option compatible with that?

Yeah, that's fine, the isolated map will apply to the rest of the uid/gid allocation.

cquike commented 1 month ago

So before changing the configuration we performed an upgrade of incus across the cluster to the latest in Debian bookworm-backports which is 6.0.1. And voila! A week after no issues anymore with network connectivity.

So maybe the issue was fixed between 6.0.0 and 6.0.1. Feel free to close this ticket then.