SSSD internal DNS resolver broken when using Cisco Anyconnect VPN client

[meskaya]

Hi everyone, I am facing an issue with SSSD internal DNS resolver (I was able to reproduce this issue with SSSD versions 2.2.3 coming with Ubuntu 20.04 and version 1.16.1 coming with Ubuntu 18.04) when I am using Cisco Anyconnect VPN client. SSSD is not detecting the new DNS servers setting up by Cisco Anyconnect client and keeps using previous ones. DNS is managed with systemd-resolved and the /etc/resolv.conf file is a symlink to /run/systemd/resolve/resolv.conf file (I am not using the internal DNS stub resolver). When Cisco Anyconnect client connects to VPN the /etc/resolv.conf symlink is renamed to /etc/resolv.conf,vpnbackup and a regular file /etc/resolve.conf is created with DNS servers to use while on VPN. When Cisco Anyconnect client disconnects from VPN the /etc/resolv.conf.vpnbackup is moved back to /etc/resolv.conf. I have checked this with inotifywait (I only kept interesting parts):

/etc/ MOVED_FROM resolv.conf
/etc/ MOVED_TO resolv.conf.vpnbackup
/etc/ CREATE resolv.conf
/etc/ OPEN resolv.conf
/etc/ ATTRIB resolv.conf
/etc/ MODIFY resolv.conf
/etc/ CLOSE_WRITE,CLOSE resolv.conf
...
/etc/ MOVED_FROM resolv.conf.vpnbackup
/etc/ MOVED_TO resolv.conf

SSSD internal resolver is then broken as SSSD fails to detect DNS changes in /etc/resolv.conf. There are no messages of Resolv.conf being updated in /var/log/sssd/sssd.log.

I can workaround this issue by changing the way SSSD detects DNS changes (stop using inotify and poll the /etc/resolv.conf file every 5 seconds) with option try_inotify set to false but I guess this can impact performance on my machine (even if I think this shouldn't be a big impact).

Is inotify not detecting a change in resolv.conf because the file is not edited but moved (to /etc/resolv.conf.vpnbackup) and a new resolv.conf file is created ?

Thanks in advance for your help :)

[alexey-tikhonov]

I think this is the same as #1031

Shall be fixed in 2.x starting 2.3.1 (0c5711f9bae1cb46d4cd3fbe5d86d8688087be13) and in 1.16.x (758b99590a8e1f69b4487fad9cf343525797e05f)

Please re-open in case you observe the same issue on a modern version.

[meskaya]

Thanks for the update @alexey-tikhonov Is there is an easy way to try modern SSSD versions on older Ubuntu releases (but LTS) ? I am doing a bug report for this issue in Ubuntu launchpad now :)

[alexey-tikhonov]

Is there is an easy way to try modern SSSD versions on older Ubuntu releases (but LTS) ?

If you want just "to try" then you can build SSSD locally from sources.

I don't know what Ubuntu policies are, but I would guess they don't do rebases in LTS versions released long ago. So I suppose your best bets are either request to backport mentioned patches or to wait 22.04 LTS

[meskaya]

For anyone facing the same issue I have raised a bug report with Ubuntu to get patches back ported to current 1.16 and 2.2.3 versions of sssd https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/1958391

[meskaya]

@alexey-tikhonov I have tested new packages the Ubuntu team built for me for Ubuntu 20.04 (available in this PPA https://launchpad.net/~paride/+archive/ubuntu/sssd-lp1958391) with https://github.com/SSSD/sssd/commit/0c5711f9bae1cb46d4cd3fbe5d86d8688087be13 patch but I still cannot see resolv.conf detection in sssd.log

~$ apt list --installed sssd*
Listing... Done
sssd-ad-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-ad/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-dbus/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed]
sssd-ipa/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-krb5-common/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-krb5/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-ldap/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-proxy/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]
sssd-tools/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed]
sssd/focal,now 2.2.3-3ubuntu0.9~paride2 amd64 [installed,automatic]

I connected to VPN and SSSD still seems unable to detect resolv.conf change :

~# ls -al /etc/resolv.conf*
-rw-r--r-- 1 root root 115 Jan 20 13:06 /etc/resolv.conf
lrwxrwxrwx 1 root root 34 Dec 8 17:30 /etc/resolv.conf.vpnbackup -> ../run/systemd/resolve/resolv.conf

# tail -f /var/log/sssd/sssd.log | egrep -i resolv

(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x4000): inotify name: .#resolv.confazayDh
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x0400): Not interested in .#resolv.confazayDh
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x4000): inotify name: .#stub-resolv.confoBQ85k
(Thu Jan 20 13:10:37 2022) [sssd] [process_dir_event] (0x0400): Not interested in .#stub-resolv.confoBQ85k

[alexey-tikhonov]

@meskaya, could you please share entire sssd.log (with debug_level=9)? Ideally with timestamps of externally observed inotifywait events.

[meskaya]

Sure no problem, please find sssd.log and inotify trace attached. I have kept sssd packages built by Ubuntu team. inotify_resolv.txt sssd.log

[alexey-tikhonov]

Could you please {stop sssd, delete old log, start sssd} instead of Received SIGHUP. Rotating logfiles. and attach the same logs?

Issue clearly happened before logs were rotated.

[alexey-tikhonov]

Ah, I didn't read initial description carefully enough.

It seems the reason is "symlink" replaced with "regular file" (as you wrote).

But inotify watch is set for a resolved name (and its dir): https://github.com/SSSD/sssd/blob/7e9269412d34e24d56d1531a2f4bbdcf5946eba4/src/util/inotify.c#L389

[_snotify_create] (0x0400): Added a watch for /run/systemd/resolve/stub-resolv.conf with inotify flags 0x8D88 internal flags 0x1 using function resolv_conf_inotify_cb after delay 1.0

Looks like a valid (unfixed) issue...

[meskaya]

New logs attached (I hope this is fine now :) ) inotify_resolv.txt sssd.log

[alexey-tikhonov]

Hm... Can't find [_snotify_create] ... Added a watch for... message in the log. Moreover, no single "inotify" related message at all.

Nonetheless, as previously mentioned, I guess the reason is a dir of a resolved symlink is watched, not "/etc/".

[alexey-tikhonov]

But there is an easy work-around (try_inotify = false). And it's not entirely clear what would be a proper fix: should we monitor both dirs/files - a symlink + resolved file..? @pbrezina, @sumit-bose, @thalman, what do you think?

[pbrezina]

I'm not entirely sure how inotify works on symlinks. Isn't IN_DONT_FOLLOW sufficient? If not, the I think we need to monitor both symlink and target.

[alexey-tikhonov]

I'm not entirely sure how inotify works on symlinks. Isn't IN_DONT_FOLLOW sufficient? If not, the I think we need to monitor both symlink and target.

Currently SSSD resolves actual file name before setting up inotify.

[meskaya]

Is there anything else I can do from my side to help you @pbrezina @alexey-tikhonov ?

[alexey-tikhonov]

Is there anything else I can do from my side to help you @pbrezina @alexey-tikhonov ?

Thank you, I think at this point we don't need additional info.

[ikerexxe]

I took a look at it and this is what I understand. On the one hand, /run/systemd/resolve/resolv.conf is the file that holds the DNS configuration and /etc/resolv.conf is a symlink to it. When the VPN client connects the /etc/resolv.conf symlink is renamed and a new file called /etc/resolv.conf is created to hold the new DNS configuration. When the VPN client disconnects the reverse process happens.

On the other hand, sssd resolves the symlinks before subscribing to any file change. From the previous explanation, this means that sssd is checking for changes in the /run/systemd/resolve/resolv.conf file, not in the /etc/resolv.conf symlink. So, if the VPN client changes /etc/resolv.conf, sssd isn't notified about it, and that's why sssd doesn't know about the new DNS settings.

[alexey-tikhonov]

I took a look at it and this is what I understand.

This is a correct summary.

[ikerexxe]

Taking that into account the only solution that comes to my mind is to stop resolving the symlinks and subscribe to the changes done to the actual file. But I guess that the name resolution is there for some reason. Do you know them?

[thalman]

For DNS resolution we use c-ares and in case of change we have to reinitialize it and resolve all the names again. With resolv.conf change some addresses might change (With VPN example to some internal IPs) or may become resolvable now or the oposite.

The /etc/resolv.conf is the source of truth. It used to be a file and on many distributions it still may be so.

In my opinion we should watch both - /etc/resolv.conf as well as its target in case of symbolic link. I hope that we will not get link pointing to link pointing to link...

[alexey-tikhonov]

But I guess that the name resolution is there for some reason. Do you know them?

This bug report is an example: some sw creates a link to the content instead putting a content directly to '/etc/resolv.conf'

In my opinion we should watch both - /etc/resolv.conf as well as its target in case of symbolic link. I hope that we will not get link pointing to link pointing to link...

+1

[aplopez]

I could reproduce this problem. I think we need to fix it.