Sometimes, physical interface does not return to host after container is stopped.

[sotiris-bos]

I am passing through physical devices. Sometimes, when I stop a container, the interface does not return to the host and there is nothing else than a reboot that fixes it.

Any idea how I can get the interface back to the host without rebooting?

[jpetazzo]

Normally, when the network namespace is deleted, the interface goes back to the host.

If the interface doesn't go back, it could be because the network namespace is not deleted.

If I understand correctly, the network namespace should be deleted when its reference count drops to zero.

The reference count corresponds to:

• how many processes are currently using that network namespace
• how many "binds" exist for the network namespace (it is possible to "save" a network namespace by doing, iirc, a mount -t bind from the pseudo file in /proc/<PID>/ns/net to something else)

So, if the interface doesn't go back, I would check:

• is there any process still running in the namespace?
• is the container really stopped?
• was there maybe another container started with --net container:... ?
• is there perhaps a monitoring tool that got a handle on the network namespace?
• any bind-mount?

[sotiris-bos]

I am sorry but I do not really know what I am doing.

I am trying to check the namespaces but I don't know how. I am running ip netns list and I am getting nothing. The container is stopped so there is no active process with the UID I have set for the application in the container.

How can I find the namespaces and PIDs?

Edit: OK, I am a bit soft in the head, I was running ip netns list on the host. I ran it in the pipework container and I got some results now. How can I identify what namespace corresponds to what container?

[jpetazzo]

Hi!

Unfortunately, "ip netns list" won't show you anything; if I remember correctly, it's only listing the content of /var/run/netns, and Docker doesn't use that.

You could check the output of "ls -l /proc/*/ns/net" and see if there are processes with different namespaces. I'm a bit tired at the moment so I can't think of another easy way but I hope this can help!

On Sat, Oct 31, 2020 at 8:00 PM sotiris-bos notifications@github.com wrote:

I am sorry but I do not really know what I am doing.

I am trying to check the namespaces but I don't know how. I am running ip netns list and I am getting nothing. The container is stopped so there is no active process with the UID I have set for the application in the container.

How can I find the namespaces and PIDs?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jpetazzo/pipework/issues/242#issuecomment-719973991, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABJ3WI6UP6QPQTSTDCU3Q3SNRNEJANCNFSM4S7FY25A .

-- @jpetazzo https://twitter.com/jpetazzo http://jpetazzo.github.io/

[sotiris-bos]

"ls -l /proc/*/ns/net" gives me a large list.

In that list, I find a process that belongs to my container user with a symlink of /proc/XXXXX/ns/net to 'net:[4026533974]' I also find three other processes under the root user with the same symlink.

If I kill these processes with kill -s 9 XXXXX XXXXX XXXXX XXXXX XXXXX they come back with different PIDs and still have the symlink to 'net:[4026533974]'. Does that mean the container did not get stopped properly?

I am running my containers with docker-compose. Is there something I can do to fix this?

I know this is not 100% related to pipework so thanks a lot for your help up to this point!

[jpetazzo]

Hi!

In that list, I find a process that belongs to my container user with a symlink of /proc/XXXXX/ns/net to 'net:[4026533974]' I also find three other processes under the root user with the same symlink.

Excellent work so far --- my interpretation is, indeed, that these three other processes are also part of the container; or at least of the network namespace of that container.

Out of curiosity, what's running in the container, and what are these 3 other processes? (You can find them in the output of ps faux, for instance.)

My guess would be, that the processes are being restarted; but I don't know exactly by what. Is there perhaps a restart policy on the container?

[sotiris-bos]

Hi!

In that list, I find a process that belongs to my container user with a symlink of /proc/XXXXX/ns/net to 'net:[4026533974]' I also find three other processes under the root user with the same symlink.

Excellent work so far --- my interpretation is, indeed, that these three other processes are also part of the container; or at least of the network namespace of that container.

Out of curiosity, what's running in the container, and what are these 3 other processes? (You can find them in the output of ps faux, for instance.)

My guess would be, that the processes are being restarted; but I don't know exactly by what. Is there perhaps a restart policy on the container?

My bad, I have two containers running with the same UID so those processes belong to the second container. If I stop my problematic container (linuxserver/plex), I cannot find any processes using the namespace. So, there is nothing under /proc/*/ns/net that is symlinked to the namespace that was being used.

The container seems to be stopped, the restart policy is "unless-stopped", no other container is using the interface and there are no symlinks under /proc/*/ns/net to the namespace.

Edit: This problem also happens with my linuxserver/transmission container. Plex and transmission are two containers that always seem to have open connections. But even if I restart NetworkManager or physically unplug the interfaces (to kill any live states) I still don't get my interfaces back to the host. Other containers do not exhibit the problem.

[jpetazzo]

OK, here are a few last ideas:

• check with sudo lsns --type net (but I don't think it can find network namespaces that are merely bind-mounted, as opposed to having active processes in them)
• grep nsfs /proc/*/mounts (but you will have to sort/filter the output!)

Or to sidestep the problem: instead of using pipework on the container running plex or transmission, start a container (I'll call it a "network container") that does nothing (you can use k8s.gcr.io/pause, tianon/sleeping-beauty, or alpine with sleep infinity), use pipework on that container, then start the other container with --net container:.... In compose you can use network_mode: "container:x" or network_mode: "service:x". Then make sure that you don't re-create that container :)

[sotiris-bos]

OK, here is a weird update:

I am in the process of moving my NAS from CentOS 8 to Arch. I finally got to ditch NetworkManager in favor of systemd-networkd.

Now on Arch, the interface is coming back to the host, but as "eth2", which is its name inside the plex container, instead of eno1v14 which is its regular name on the host.

I am using predictable network interface names on Arch (as I was on CentOS), not sure if that has anything to do with it. Anyway, I now have a solution of renaming the interface via ip link and restarting the container, not having to fully reboot the host.

[jpetazzo]

Yay, that's great news!

If the name is changed by pipework, this should be relatively easy to comment out the code that does it so that it remains the same (if that's what you want). If you want the name to be ethX in the container, but change back when the interface returns to the host - that might be more complicated :thinking: