[Error] resolv.conf is not updated when it changes on the host

[peterurbanec]

When /etc/resolv.conf changes on the host system, the corresponding file is not updated in the running distrobox image.

Host system is OpenSUSE Tumbleweed:

NAME="openSUSE Tumbleweed"
# VERSION="20220802"

Using distrobox 1.3.1 with docker:

$ rpm -q distrobox
distrobox-1.3.1-2.1.noarch
$ rpm -q docker
docker-20.10.17_ce-2.1.x86_64

Running on a btrfs filesystem configuration as per OpenSUSE installation defaults.

To Reproduce

distrobox-create --image quay.io/centos/centos:7 --name test0
distrobox-enter test0
cat /etc/resolv.conf

Change contents of /etc/resolv.conf on the host system (for example by initiating a VPN connection) and wait about a minute to ensure that there was sufficient time for the update to propagate (code suggest that the refresh is attempted every 15 seconds).

distrobox-enter test0
cat /etc/resolv.conf

Observe that /etc/resolv.conf has not been updated to match the host.

Logs docker logs shows the following errors constantly repeating:

mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/localtime: not a directory
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/localtime: not a directory

[89luca89]

@peterurbanec

The error you see here:

mountpoint: ...: not a directory\

Is ok, it's just a warning, anyway I cannot reproduce this on my system (OpenSUSE MicroOS + Podman) nor on a TW nor an Ubuntu system with docker sadly

[89luca89]

Any updates on this?

[peterurbanec]

I set up a fresh instance of quay.io/centos/centos:7 image under Tumbleweed+docker and I still experience the same problem. If I distrobox-enter before the VPN is up, then start VPN on host side, the running distrobox instance continues to use the old /etc/resolv.conf

I noticed that if I sudo inside the centos7 distrobox, I can not do anything with the /etc/resolv.conf file mounted from the host. I can not edit it or unmount it. Maybe that's expected.

Is there some extra information I could collect to help narrow this down?

[89luca89]

@peterurbanec which type of VPN?

With openvpn (using networkmanager) I can't reproduce this

[peterurbanec]

I'm using FortiSSL VPN. The relevant packages are openfortivpn, NetworkManager-fortisslvpn and plasma-nm5-fortisslvpn.

[peterurbanec]

The VPN software on the host system updates the configuration like this:

Before VPN activation:

# ls -al /etc/resolv.conf /run/netconfig/resolv.conf
lrwxrwxrwx 1 root root 26 Feb 9 2022 /etc/resolv.conf -> /run/netconfig/resolv.conf
-rw-r--r-- 1 root root 646 Aug 23 11:14 /run/netconfig/resolv.conf

After VPN activation:

# ls -al /etc/resolv.conf /run/netconfig/resolv.conf
lrwxrwxrwx 1 root root 26 Feb 9 2022 /etc/resolv.conf -> /run/netconfig/resolv.conf
-rw-r--r-- 1 root root 696 Aug 23 11:56 /run/netconfig/resolv.conf

[89luca89]

@peterurbanec hey I've pushed an update today that fixes stuff about symlinks handling for the sync, can you check if it fixes also your problem? Thanks!

[89luca89]

@peterurbanec any news?

[peterurbanec]

My apologies, I have not had a chance to test yet. I will try to do so in the next week.

It looks like the OpenSUSE packages are still at version 1.3.1, so I will have to get the git version installed. Would you like me to test any specific tag or is the HEAD commit on main branch the best version to test?

[89luca89]

Hi @peterurbanec

you can use the install script to install the latest release, it will have the fix to make it work, you will have to stop, and start again the containers after you've updated distrobox

[peterurbanec]

I tested with the openSUSE packaged version of 1.4.0 and also with main branch from git. Neither version works at all.

Running distrobox-enter results in:

Error response from daemon: mount /:/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host, flags: 0x5000: no space left on device

and the creation of about nine thousand mount entries on the host, similar to these:

tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/3142c48bd5d2ad3353af0a96b28e58b365eef767f7ad9f86e8ace7d69a234270/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)
tmpfs on /var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/run/host/var/lib/docker/containers/857d3dec059d3638772459387d02a1abb55aeb804f32e30522ca75725dc2cfe2/mounts/secrets type tmpfs (ro,nosuid,nodev,noexec,relatime,inode64)

Going back to git tag 1.3.1 resolves the mount issue and I can start the same distrobox container without issues.

[peterurbanec]

I have not found a way of removing the problematic mount entries from the host side, other than rebooting the machine.

[89luca89]

Seems related to this #399

Need to investigate

[peterurbanec]

I have tested this with 1.4.1 and I'm still not seeing updates in the distrobox container when the host /etc/resolv.conf contents change.

On the host:

peteru@leap:~$ ls -al /etc/resolv.conf /run/netconfig/resolv.conf
lrwxrwxrwx 1 root root  26 Aug 26 02:08 /etc/resolv.conf -> /run/netconfig/resolv.conf
-rw-r--r-- 1 root root 646 Sep 19 14:42 /run/netconfig/resolv.conf

In distrobox:

peteru@centos7:~$ ls -al /etc/resolv.conf
-rw-r--r-- 1 root root 646 Sep 19 14:56 /etc/resolv.conf
peteru@centos7:~$ mount | grep -i resolv
tmpfs on /etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)

Next, I connect to VPN using Network Manager GUI from KDE and examine the same files again. I can see a change on the host, but the distrobox container has not changed:

Host:

peteru@leap:~$ ls -al /etc/resolv.conf /run/netconfig/resolv.conf
lrwxrwxrwx 1 root root  26 Aug 26 02:08 /etc/resolv.conf -> /run/netconfig/resolv.conf
-rw-r--r-- 1 root root 696 Sep 19 14:59 /run/netconfig/resolv.conf

Distrobox:

peteru@centos7:~$ ls -al /etc/resolv.conf
-rw-r--r-- 0 root root 646 Sep 19 14:56 /etc/resolv.conf
peteru@centos7:~$ mount | grep -i resol
tmpfs on /etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)

Docker log:

+ printf 'distrobox: Executing init hooks...\n'
+ eval
+ '[' 0 -eq 0 ']'
+ printf 'container_setup_done\n'
+ HOST_WATCH='
                /etc/hosts
                /etc/localtime
                /etc/resolv.conf
        '
+ set +x
container_setup_done
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/resolv.conf: not a directory
mountpoint: /etc/resolv.conf: not a directory

[89luca89]

Another thing, can you at this point check:

PRE-vpn:

sha256sum /etc/resolv.conf

both inside and outside

Then AFTER vpn:

sha256sum /etc/resolv.conf inside and outside, and also sha256sum /run/host/etc/resolv.conf

At this point I'm suspecting that the file inside the host's FS I mount, which is the reference I use to keep file in sync, is not updated accordingly when you connect to the vpn

On my side this is working for me on openvpn

[Minkiu]

Hello, sorry to barge into this issue, but I am trying to achieve something similar, I want to run openfortivpn from within the container (I'm on an immutable OS), but I'm getting from openfortivpn:

Couldn't open the /dev/ppp device: Permission denied
/usr/sbin/pppd: Sorry - this system lacks PPP kernel support

ERROR:  read: Input/output error
INFO:   Cancelling threads...
ERROR:  pppd: The kernel does not support PPP, for example, the PPP kernel driver is not included or cannot be loaded.

I tried creating the box as follow: distrobox create -n vpn-box -i ubuntu:20.04 -a "--device /dev/ppp" but no luck,

By looking on the inet, --device /dev/ppp seems to be required, but at podman run, which as far as I could tell distrobox doesn't use, and podman exec does not accept the above flag...

I guess being able to run the VPN within the container would (tangentially) solve the OP issue...

Happy to create another issue if you prefer!

Thanks for the project!

[peterurbanec]

@Minkiu

I do not think that your issue is related to this one. You may be better off creating a new issue for your particular scenario so that it can get proper attention.

[peterurbanec]

Before VPN

Outside:

peteru@leap:~$ sha256sum /etc/resolv.conf
f3bc45fae1e1f96db2b3b3131280242677d3359bbacf95fec3811f5724832202  /etc/resolv.conf

Inside:

peteru@centos7:~$ sha256sum /etc/resolv.conf
f3bc45fae1e1f96db2b3b3131280242677d3359bbacf95fec3811f5724832202  /etc/resolv.conf

peteru@centos7:~$ sha256sum /run/host/etc/resolv.conf
f3bc45fae1e1f96db2b3b3131280242677d3359bbacf95fec3811f5724832202  /run/host/etc/resolv.conf

After VPN

Outside:

peteru@leap:~$ sha256sum /etc/resolv.conf
408d84ba8d11a2de6715434cea2c98bccc03eaa0cf208879c5b634f01ca53b40  /etc/resolv.conf

Inside:

peteru@centos7:~$ sha256sum /etc/resolv.conf
f3bc45fae1e1f96db2b3b3131280242677d3359bbacf95fec3811f5724832202  /etc/resolv.conf

peteru@centos7:~$ sha256sum /run/host/etc/resolv.conf
408d84ba8d11a2de6715434cea2c98bccc03eaa0cf208879c5b634f01ca53b40  /run/host/etc/resolv.conf

[89luca89]

@peterurbanec

aha! There lies the issue :smile:

Seems like openfortivpn is doing something strane on /etc/resolv.conf, so it is updated on the host, but its does not reflect in /run/host, which is the host's filesystem mounted inside the container

That's why the file is not picked up... not sure why, can you check after you're connected to the vpn the command mountpoint /etc/resolv.conf on the host?

[peterurbanec]

I think /run/host/etc/resolv.conf is being updated (the checksum matches the host side), but the change is not being propagated to /etc/resolv.conf. The message mountpoint: /etc/resolv.conf: not a directory might be the clue here.

On host before VPN

peteru@leap:~$ ls -al /etc/resolv.conf
lrwxrwxrwx 1 root root 26 Aug 26 02:08 /etc/resolv.conf -> /run/netconfig/resolv.conf
peteru@leap:~$ mountpoint /etc/resolv.conf
/etc/resolv.conf is not a mountpoint
peteru@leap:~$ mountpoint /run/netconfig/resolv.conf
/run/netconfig/resolv.conf is not a mountpoint

In container before VPN

peteru@centos7:~$ ls -al /etc/resolv.conf 
-rw-r--r-- 1 root root 646 Sep 29 12:20 /etc/resolv.conf
peteru@centos7:~$ mountpoint /etc/resolv.conf 
mountpoint: /etc/resolv.conf: not a directory

peteru@centos7:~$ mount | grep resolv\.conf
tmpfs on /etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)

On host after VPN

peteru@leap:~$ mountpoint /etc/resolv.conf
/etc/resolv.conf is not a mountpoint
peteru@leap:~$ mountpoint /run/netconfig/resolv.conf
/run/netconfig/resolv.conf is not a mountpoint

In container after VPN

peteru@centos7:~$ mountpoint /etc/resolv.conf
mountpoint: /etc/resolv.conf: not a directory
peteru@centos7:~$ mount | grep resolv\.conf
tmpfs on /etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)
tmpfs on /run/host/var/lib/docker/btrfs/subvolumes/5edda9c2f6fcab0886537c730b967b4128662c7764ec1274d1d35ae0945f44e0/etc/resolv.conf type tmpfs (ro,size=26335504k,nr_inodes=819200,mode=755,inode64)

It seems the centos7 mountpoint command can not deal with a tmpfs mounted file and instead expects a mount point to be a directory.

[peterurbanec]

I tested with a modification to distrobox-init commenting out the following line:

mountpoint "${file_watch}" &&

to remove the mountpoint check.

This fixes the issue and the resolv.conf inside the container now seems to track changes on the host. :smile:

[89luca89]

@peterurbanec

if the mountpoint is the problem, can you check if using another distribution (say, centos8 or fedora, or ubuntu) inside the distrobox will indeed work?

Thanks!

[89luca89]

@peterurbanec finally I can reproduce the bug :smile:

I've fixed it in the latest commit, simply moving from mountpoint to findmnt, and it seems to be able to detect file mounts just fine :+1:

[89luca89]

To make it work update to latest git version stop the container, restart it again

should be all working now

[alexbradd]

Looks like this bug came back. Here is my environment:

• Distrobox git, podman 4.7.2 running on the latest opensuse Tumbleweed
• Container image is registry.opensuse.org/opensuse/distrobox:latest (hash 7463f9a4f5a0)
• Container has been created using distrobox assemble and no interesting options (some extra packages and an init script that copies some files from my home directory)
• Root filesystem is luks + BTRFS

What happens is the same thing: on network changes files in HOST_WATCH do not get synced in with the host. I did some digging the reason is that for some reason all mounts are of type btrfs instead of the expected overlay.

Restarting the container fixes the issue.

[alexbradd]

Retested with latest distrobox from main and a fresh container (registry.opensuse.org/opensuse/distrobox:latest) and this is still a problem. Disk is still on luks + BTRFS and the problem with HOST_WATCH is still the same.

Ping @89luca89 to maybe reopen the issue

[lucas-stauder]

I'm having the same issue on Vanilla OS 2