canonical / ubuntu-pro-client

Ubuntu Pro Client for offerings from Canonical
https://canonical-ubuntu-pro-client.readthedocs-hosted.com/en/latest/
GNU General Public License v3.0
52 stars 73 forks source link

Bug: apparmor denies /var/lib/dpkg/arch #3137

Closed panlinux closed 3 months ago

panlinux commented 4 months ago

Description of the bug

Seen in the logs from https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2067728:

May 31 18:00:17 loky kernel: audit: type=1400 audit(1717158617.647:261): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8510 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:17 loky kernel: audit: type=1400 audit(1717158617.678:262): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8511 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:17 loky kernel: audit: type=1400 audit(1717158617.680:263): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8512 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:17 loky kernel: audit: type=1400 audit(1717158617.683:264): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8513 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:18 loky kernel: audit: type=1400 audit(1717158618.556:265): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8712 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:18 loky kernel: audit: type=1400 audit(1717158618.601:266): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8714 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:18 loky kernel: audit: type=1400 audit(1717158618.603:267): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8715 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:18 loky kernel: audit: type=1400 audit(1717158618.607:268): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8720 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
May 31 18:00:18 loky kernel: audit: type=1400 audit(1717158618.622:269): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=8725 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Unsure if that's the cause of the LP bug, though, as the ua logs also show errors contacting the contract server (timeouts).

The journal logs at those times are fine:

May 31 18:00:17.176632 loky systemd[1]: Starting apt-news.service - Update APT News...
May 31 18:00:17.184741 loky systemd[1]: Starting esm-cache.service - Update the local ESM caches...
May 31 18:00:17.370505 loky systemd[1]: apt-news.service: Deactivated successfully.
May 31 18:00:17.370797 loky systemd[1]: Finished apt-news.service - Update APT News.
May 31 18:00:18.673192 loky systemd[1]: esm-cache.service: Deactivated successfully.
May 31 18:00:18.673539 loky systemd[1]: Finished esm-cache.service - Update the local ESM caches.
May 31 18:00:18.673778 loky systemd[1]: esm-cache.service: Consumed 1.385s CPU time.
May 31 18:05:58.552545 loky systemd[1]: Starting ua-timer.service - Ubuntu Pro Timer for running repeated jobs...
May 31 18:05:58.739150 loky systemd[1]: ua-timer.service: Deactivated successfully.
May 31 18:05:58.739422 loky systemd[1]: Finished ua-timer.service - Ubuntu Pro Timer for running repeated jobs.

Incidentally, looks like apport is trying to report the version of the ubuntu-advantage-tools package, but it should also check ubuntu-pro-client.

Expected behavior

No apparmor denied errors.

Current behavior For some unknown operation, the pro client on that system is triggering dpkg apparmor denied errors. We haven't seen those in our testing.

in uaclient/system.py, we have get_dpkg_arch() which calls dpkg --print-architecture, and that works just fine with the current apparmor profile:

# aa-exec -p ubuntu_pro_esm_cache//dpkg dpkg --print-architecture
amd64

And strace confirms that that command does not touch /var/lib/dpkg:

# strace -f dpkg --print-architecture 2>&1|grep open
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libmd.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/etc/dpkg/dpkg.cfg.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
openat(AT_FDCWD, "/etc/dpkg/dpkg.cfg", O_RDONLY) = 3
openat(AT_FDCWD, "/root/.dpkg.cfg", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/share/locale/en_US.UTF-8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US.utf8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en_US/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.UTF-8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en.utf8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale/en/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.UTF-8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US.utf8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en_US/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en.UTF-8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en.utf8/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/share/locale-langpack/en/LC_MESSAGES/dpkg.mo", O_RDONLY) = -1 ENOENT (No such file or directory)

To Reproduce Unknown at the moment.

System information:

Additional context

Add any other context about the problem here.

panlinux commented 4 months ago

Ok, had an excellent troubleshoting session with @renanrodrigo and we found the origin of the problem.

Basically, just the presence of /var/lib/dpkg/arch is enough to trigger the attempted access and subsequent DENIED. Even if the file is empty. But the most common case seems to be systems that have a subarchitecture, like i386, added like dpkg --add-architecture i386. That will create /var/lib/dpkg/arch with amd64 and i386 in it.

We don't know yet the consequence of this DENIED error. It looks like it's triggered by apt-cache policy called by Pro, which ends up calling dpkg --print-foreign-architectures, which is what attempts to read /var/lib/dpkg/arch.

I diffed the output of apt-cache policy with and without the apparmor profile, and even though the run with the apparmor profile had the DENIED log entries, the actual output has no differences:

$ sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy > denied
$ sudo apt-cache policy > allowed
$ diff -u allowed denied
$

It's the same attached or unattached.

panlinux commented 4 months ago

Further troubleshooting shows that also the apt_methods_gpgv child profile is affected in this situation:

[Mon Jun  3 13:39:19 2024] audit: type=1400 audit(1717421960.564:105): apparmor="DENIED" operation="open" profile="ubuntu_pro_esm_cache//apt_methods_gpgv" name="/var/lib/dpkg/arch" pid=4879 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=105 ouid=0

I'll go over the profiles and make sure /var/lib/dpkg is allowed whenever we are dealing with apt or dpkg.

dominicraf commented 3 months ago

Confirm that I have this problem on 2 machines running bionic which have empty /var/lib/dpkg/arch

renanrodrigo commented 3 months ago

@dominicraf ok that is interesting The presence of the file was, for us, a guaranteed way to trigger the bug, but of course other situations may cause it. The fix we applied (and are in process to release) will cover any case by fixing the apparmor profile, but I'm curious - what happens when you run sudo aa-exec -p ubuntu_pro_esm_cache apt-cache policy ?

panlinux commented 3 months ago

Confirm that I have this problem on 2 machines running bionic which have empty /var/lib/dpkg/arch

That statement is a bit ambiguous. Do you have a /var/lib/dpkg/arch file that is empty, or do you not have that file?

renanrodrigo commented 3 months ago

Aah yes good catch - the presence of the file itself is enough, content won't matter

dominicraf commented 3 months ago

Sorry, correction, and apologies for any confusion, my comment was rather careless. I am running jammy (22.04), not bionic, and the file /var/lib/dpkg/arch exists and has 2 lines as contents:

# cat /var/lib/dpkg/arch
amd64
i386

The following returns nothing (so it seems I actually have no i386 packages): # dpkg -l | awk '/^ii/ && $4 == "i386" { print }'

And in answer to the q above:

# aa-exec -p ubuntu_pro_esm_cache apt-cache policy
Package files:
 100 /var/lib/dpkg/status
     release a=now
 510 https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates/main amd64 Packages
     release v=22.04,o=UbuntuESM,a=jammy-infra-updates,n=jammy,l=UbuntuESM,c=main,b=amd64
     origin esm.ubuntu.com
 510 https://esm.ubuntu.com/infra/ubuntu jammy-infra-security/main amd64 Packages
     release v=22.04,o=UbuntuESM,a=jammy-infra-security,n=jammy,l=UbuntuESM,c=main,b=amd64
     origin esm.ubuntu.com
 500 https://packagecloud.io/ookla/speedtest-cli/ubuntu jammy/main amd64 Packages
     release v=1,o=packagecloud.io/ookla/speedtest-cli,a=jammy,n=jammy,l=speedtest-cli,c=main,b=amd64
     origin packagecloud.io
 500 https://ppa.launchpadcontent.net/maxmind/ppa/ubuntu jammy/main amd64 Packages
     release v=22.04,o=LP-PPA-maxmind,a=jammy,n=jammy,l=MaxMind Libraries and Software,c=main,b=amd64
     origin ppa.launchpadcontent.net
 500 http://ppa.launchpad.net/maxmind/ppa/ubuntu jammy/main amd64 Packages
     release v=22.04,o=LP-PPA-maxmind,a=jammy,n=jammy,l=MaxMind Libraries and Software,c=main,b=amd64
     origin ppa.launchpad.net
 500 https://ppa.launchpadcontent.net/adiscon/v8-stable/ubuntu jammy/main amd64 Packages
     release v=22.04,o=LP-PPA-adiscon-v8-stable,a=jammy,n=jammy,l=rsyslog v8-stable,c=main,b=amd64
     origin ppa.launchpadcontent.net
 500 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu jammy/main amd64 Packages
     release v=22.04,o=LP-PPA-adiscon-v8-stable,a=jammy,n=jammy,l=rsyslog v8-stable,c=main,b=amd64
     origin ppa.launchpad.net
 500 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=multiverse,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=universe,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=restricted,b=amd64
     origin security.ubuntu.com
 500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-security,n=jammy,l=Ubuntu,c=main,b=amd64
     origin security.ubuntu.com
 100 http://gb.archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-backports,n=jammy,l=Ubuntu,c=universe,b=amd64
     origin gb.archive.ubuntu.com
 100 http://gb.archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-backports,n=jammy,l=Ubuntu,c=main,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-updates,n=jammy,l=Ubuntu,c=multiverse,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-updates,n=jammy,l=Ubuntu,c=universe,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-updates,n=jammy,l=Ubuntu,c=restricted,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy-updates,n=jammy,l=Ubuntu,c=main,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy/multiverse amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy,n=jammy,l=Ubuntu,c=multiverse,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy/universe amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy,n=jammy,l=Ubuntu,c=universe,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy/restricted amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy,n=jammy,l=Ubuntu,c=restricted,b=amd64
     origin gb.archive.ubuntu.com
 500 http://gb.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
     release v=22.04,o=Ubuntu,a=jammy,n=jammy,l=Ubuntu,c=main,b=amd64
     origin gb.archive.ubuntu.com
Pinned packages:
#
eugenesan commented 3 months ago

Until the next release, the fix is offered at https://bugs.launchpad.net/ubuntu/+source/ubuntu-advantage-tools/+bug/2067810

Just install ubuntu-pro-client (32.3.1~24.04) from Proposed.