Error setting extended attributes on "/catatonit"

AmedeeBulle commented 1 year ago

Issue Description

This is the same issue as described in #18064, however I discovered that it is not specific to Oracle Linux, It can easily be reproduced in RHEL 9.1 and 9.2.

When IMA signature is present on the catatonit executable (/usr/libexec/catatonit/catatonit for RHEL 9.1 /usr/libexec/podman/catatonit for RHEL 9.2), one cannot create a rootless pod (see error below).

IMA signatures are set

at package installation time
if the rpm-plugin-ima package is already installed

It seems that rpm-plugin-ima isn't installed by default on RHEL systems, which makes that you don't always run into the issue.

As far as I can see, the copy of IMA attributes has been introduced by containers/storage#657. Since the IMA attributes can only be copied as root, shouldn't we skip the copy (or ignore EPERM error) in rootless mode?

Steps to reproduce the issue

[opc@localhost ~]$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.2 (Plow)
[opc@localhost ~]$ sudo -i
<SNIP>
[root@localhost ~]# # No IMA signature present:                                                                                                                                     
[root@localhost ~]# getfattr -d -m - /usr/libexec/podman/catatonit                                                                                                                                     
getfattr: Removing leading '/' from absolute path names                                                                                                                                                  
# file: usr/libexec/podman/catatonit                                                                                                                                                                     
security.selinux="system_u:object_r:bin_t:s0"                                             

[root@localhost ~]# # We install the IMA rpm plugin and re-install podman:
[root@localhost ~]# dnf install -y rpm-plugin-ima
<SNIP>
[root@localhost ~]# dnf reinstall -y podman
<SNIP>

[root@localhost ~]# # We now have an IMA signature
[root@localhost ~]# getfattr -d -m - /usr/libexec/podman/catatonit
getfattr: Removing leading '/' from absolute path names
# file: usr/libexec/podman/catatonit
security.ima=0sAwIE0zIESQBmMGQCMDQAp/DO/kGOV2pgd0dGOw4EbsxHnHXANwqILX4vNyTjfB78qFGMxqQTn9BPkT5H3AIwdJ8wDHyQGSTSI5Fi3ABthWXej6gPyHUwXyElI1ysvaEntPFAfe+4cA8pjNSqtW87
security.selinux="system_u:object_r:bin_t:s0"
[root@localhost ~]# 
logout

[opc@localhost ~]$ # As user, try to create a pod
[opc@localhost ~]$ podman pod create test
Error: building local pause image: building at STEP "COPY /usr/libexec/podman/catatonit /catatonit": storing "/usr/libexec/podman/catatonit": error during bulk transfer for copier.request{Request:"PUT", Root:"/", preservedRoot:"/home/opc/.local/share/containers/storage/overlay/0adae3dcb1489aaa997f3706f3ab8e8d101c3e7fd0598b33189a877f8a3200d5/merged", rootPrefix:"/home/opc/.local/share/containers/storage/overlay/0adae3dcb1489aaa997f3706f3ab8e8d101c3e7fd0598b33189a877f8a3200d5/merged", Directory:"/", preservedDirectory:"/home/opc/.local/share/containers/storage/overlay/0adae3dcb1489aaa997f3706f3ab8e8d101c3e7fd0598b33189a877f8a3200d5/merged", Globs:[]string{}, preservedGlobs:[]string{}, StatOptions:copier.StatOptions{CheckForArchives:false, Excludes:[]string(nil)}, GetOptions:copier.GetOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), Excludes:[]string(nil), ExpandArchives:false, ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, KeepDirectoryNames:false, Rename:map[string]string(nil), NoDerefSymlinks:false, IgnoreUnreadable:false, NoCrossDevice:false}, PutOptions:copier.PutOptions{UIDMap:[]idtools.IDMap{}, GIDMap:[]idtools.IDMap{}, DefaultDirOwner:(*idtools.IDPair)(0xc0005dddb0), DefaultDirMode:(*fs.FileMode)(nil), ChownDirs:(*idtools.IDPair)(nil), ChmodDirs:(*fs.FileMode)(nil), ChownFiles:(*idtools.IDPair)(nil), ChmodFiles:(*fs.FileMode)(nil), StripSetuidBit:false, StripSetgidBit:false, StripStickyBit:false, StripXattrs:false, IgnoreXattrErrors:false, IgnoreDevices:true, NoOverwriteDirNonDir:false, NoOverwriteNonDirDir:false, Rename:map[string]string(nil)}, MkdirOptions:copier.MkdirOptions{UIDMap:[]idtools.IDMap(nil), GIDMap:[]idtools.IDMap(nil), ChownNew:(*idtools.IDPair)(nil), ChmodNew:(*fs.FileMode)(nil)}, RemoveOptions:copier.RemoveOptions{All:false}}: copier: put: error setting extended attributes on "/catatonit": setting value of extended attribute "security.ima" on "/catatonit": operation not permitted
[opc@localhost ~]$

Describe the results you received

See above

Describe the results you expected

A pod created

podman info output

[opc@localhost ~]$ podman info                                                                                                                                                                   [72/1645]
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 606c693de21bcbab87e31002e46663c5f2dc8a9b'
  cpuUtilization:
    idlePercent: 98.88
    systemPercent: 0.51
    userPercent: 0.61
  cpus: 1
  distribution:
    distribution: '"rhel"'
    version: "9.2"
  eventLogger: journald
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.14.0-284.11.1.el9_2.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 3217227776
  memTotal: 3838230528
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:                                                                                                                                                                                  [17/1645]
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 3439325184
  swapTotal: 3439325184
  uptime: 0h 34m 11.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/opc/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/opc/.local/share/containers/storage
  graphRootAllocated: 29827903488
  graphRootUsed: 1507196928
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/opc/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1676978277
  BuiltTime: Tue Feb 21 12:17:57 2023
  GitCommit: ""
  GoVersion: go1.19.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

Luap99 commented 1 year ago

@AmedeeBulle This is properly better handled by going through the Red Hat channels (i.e. bugzilla)

In any case, @giuseppe @nalind WDYT?

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

cgwalters commented 1 year ago

I think the fix here is to ensure that buildPauseImage strips all xattrs from the binary.

(Side note: I found it non-obvious that because ContextDirectory is unset that it defaults to / from the host)

Anyways in the end, I think https://github.com/containers/storage/pull/657 has introduced a new problem in that security.ima started being propagated by default, but then we need to turn that off for cases where it's not desired.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

michaeloa commented 11 months ago

Is there a workaround for this issue?

AmedeeBulle commented 11 months ago

Is there a workaround for this issue?

If you don't need IMA, remove the rpm-plugin-ima package and reinstall podman.

Alternatively, build the pause container as root, save it and load it as the target user.

Luap99 commented 4 months ago

I think another work around would be to use the k8s pause image instead, i.e. set this in containers.conf

[engine]
infra_image="registry.k8s.io/pause:latest"

ikke-t commented 3 months ago

I was hit with the same error on fedora-iot, where it's not convenient to remove and re-install packages. The above k8s pause made me get running.

edsantiago commented 3 months ago

And now seeing this on rawhide rootless

edsantiago commented 3 months ago

As of 2024-06-07, ima-evm-utils-1.5-5.fc41 now Requires rpm-plugin-ima. This prevents rootless podman from working.

I could file a bz against ima-evm-utils, but that seems whiny: it's their package, they can require whatever they want, I think it's our problem if we can't deal with the presence of an installed RPM.

@containers/podman-maintainers PTAL, this is likely to blow up in rawhide.

Luap99 commented 3 months ago

As of 2024-06-07, ima-evm-utils-1.5-5.fc41 now Requires rpm-plugin-ima. This prevents rootless podman from working.

I could file a bz against ima-evm-utils, but that seems whiny: it's their package, they can require whatever they want, I think it's our problem if we can't deal with the presence of an installed RPM.

@containers/podman-maintainers PTAL, this is likely to blow up in rawhide.

Do you see the attribute with getfattr -d -m security.ima /usr/libexec/catatonit/catatonit? I installed the mentioned versions above but don't see any ima signatures

[root@vm-10-0-185-1 ~]# rpm -q ima-evm-utils
ima-evm-utils-1.5-5.fc41.x86_64
[root@vm-10-0-185-1 ~]# rpm -q rpm-plugin-ima
rpm-plugin-ima-4.19.91-8.fc41.x86_64
[root@vm-10-0-185-1 ~]# getfattr -d -m security.ima /usr/libexec/catatonit/catatonit 
[root@vm-10-0-185-1 ~]#

Of course I agree that podman shouldn't fail and ignore IMA for this case.

And from a quick search I couldn't find any fedora change proposals that mention IMA

edsantiago commented 3 months ago

getfattr -d -m security.ima /usr/libexec/catatonit/catatonit 
getfattr: Removing leading '/' from absolute path names
# file: usr/libexec/catatonit/catatonit
security.ima=0sAwIEh210NQBHMEUCIDZQ7GwaonjcclDCtlCHb4WkWhiPa/a+f/LTQBihDoDxAiEA7IA+rx6mANRGnM7BNSWm34mB2N5BpeiWvKDzwOrfZ/k=

Maybe order matters? Need to install podman+catatonit after rpm-plugin-ima?

Reproducer:

rawhide vm (1mt)
dnf upgrade, reboot (necessary to get newer systemd, which fixes the rootless XDG bug)
dnf install podman
loginctl enable-linger fedora; su - fedora; podman run --pod new:foo quay.io/libpod/testimage:20240123 date (barf)

[edit: update: setfattr -x security.ima /usr/libexec/catatonit/catatonit makes rootless podman work again]

cgwalters commented 3 months ago

I'd argue for a revert of https://github.com/containers/storage/pull/657 though it may be painful at this point...do we know if there are consumers of that? I feel like the use case of injecting IMA into a container build is a "lower level API" that should be opt-in. This could be something like buildah from --enable-ima or so?

Maybe order matters? Need to install podman+catatonit after rpm-plugin-ima?

Yes, almost certainly.

Luap99 commented 3 months ago

Well the whole thing was added because a user wanted IMA to be copied (https://github.com/containers/buildah/issues/2127) so there are almost certainly users that depend on it. So reverting is a breaking change and we cannot justify that in a minor release IMO.

Also I am not sure how IMA works but couldn't a policy just deny all files without the stored hash or is this not a thing? In this case dropping the signatures would be bad too.

Also overall it seems to work fine as root so chaining existing behaviour is undesirable. Of course it doesn't work rootless so maybe the easiest non breaking change is to ignore the attribute when running rootless but I am not a storage maintainer so it is more important what they think.

@mtrmac @giuseppe @nalind

giuseppe commented 3 months ago

couldn't we just ignore the error when setting the IMA xattr fails with EPERM?

mtrmac commented 3 months ago

TL:DR: could we special-case the buildPauseImage code to strip the IMA attribute?

Or would that be insufficient because ordinary users tend to do COPY /usr/bin/… in their build processes as well?

If the latter, I think COPY should drop the IMA attribute (when rootless? always?). That’s not helpful to users who actually want IMA, but I don’t think it provides actually valuable security, so I’m not inclined to support a large investment to “virtualize” the IMA attributes so that non-root users can build images with IMA attributes.

So the situation is that default rawhide systems might be storing IMA signatures (based on data distributed inside RPMs), but they are not enforcing signatures to be present when files are used?

In the specific case of the pause image … I’m tempted to say that locally building an image at runtime, when that could happen at distribution build time is clearly suboptimal (or we might not need to create a container image just to be able to create a container, when podman create --rootfs exists), but, also, probably, out of scope of this issue — and I don’t know what I don’t know, I’m sure buildPauseImage was added for a good reason..

If IMA were enforcing:

First, AFAIK IMA does not authenticate directories, and there are known exploits of that. I don’t know what enabling it buys the user. I’d strongly recommend a more comprehensive authentication mechanism, like dm-verity.

My understanding (per https://github.com/containers/storage/pull/1608#issuecomment-1557551200 ) is that in general, for rootless pulls / builds we are in a mostly impossible situation with IMA (given current kernel feature set).

Also, none of the popular images contain embedded IMA signatures. So we would end up with a container host only being able to run specially-built images. On net, given the weak security value of IMA, and the extra steps necessary to make it enforcing, I think I’m fine with also asking users to set infra_image manually.

Luap99 commented 1 month ago

Friendly ping that this still needs someone to fix it. We added a work around in our CI images but it is really not sustainable that many podman users have to apply a workaround in order to use pods.

mheon commented 1 month ago

We have this on the agenda for the cabal tomorrow BTW

edsantiago commented 1 month ago

Ping, rawhide is close to releasing, and this is still not fixed. Podman users on f41 may be in for a very unpleasant surprise.

mheon commented 1 month ago

We decided to work on removing the pause image entirely, but that sounds like it will take time we may not have. Is there a temporary workaround we can use while that work gets scheduled?

gucci-on-fleek commented 1 month ago

@mheon

Is there a temporary workaround we can use while that work gets scheduled?

This issue is affecting me right now on Fedora IoT 40 (x86_64), but @Luap99's suggestion fixed it for me:

I think another work around would be to use the k8s pause image instead, i.e. set this in containers.conf
[engine]
infra_image="registry.k8s.io/pause:latest"

So one option for a temporary workaround would be to add a pause image to registry.fedoraproject.org, then add the following to /usr/share/containers/containers.conf:

[engine]
infra_image="registry.fedoraproject.org/pause:41"

This does mean that you would need an internet connection the first time that you launch a container, so another option would be to somehow include the image directly in the podman package. Maybe you can put it somewhere in /usr/lib/containers/storage/?

rhatdan commented 1 month ago

You could also pre-pull the pause image into containers-storage, for at least the rootful case.

rhatdan commented 1 month ago

If you are using a bootc or image mode deployment.

edsantiago commented 1 month ago

Removing security.ima from all vendored code (buildah, c/storage) works

mheon commented 1 month ago

I think we want to make it selective (root still gets IMA, rootless does not) but I think that's our way forward.

cgwalters commented 1 month ago

My proposal would be to first make this a storage.conf option like xattrs_to_serialize = ["user.*", "security.capability", "security.ima" ]" or so: this would allow everyone who doesn't want security.ima leaking into their containers just because they happen to have rpm-plugin-ima installed to avoid it, i.e. to manually revert the original change by setting xattrs_to_serialize = ["user.*", "security.capability"].

Then the internal pause image build flips on that storage option.

nalind commented 1 month ago

The storage library isn't the only place where we read and write those (as @edsantiago noticed, we have logic in buildah that also handles these there), and I'm not generally fond of the idea of having a configuration setting that needs to be set one way for root and one way for everyone else.

cgwalters commented 1 month ago

The storage library isn't the only place where we read and write those (as @edsantiago noticed, we have logic in buildah that also handles these there),

Hmm but once we had the option, buildah could read it from c/storage right?

and I'm not generally fond of the idea of having a configuration setting that needs to be set one way for root and one way for everyone else.

Note I'm not currently suggesting that the default value for this would change depending on root vs not; it's more targeting the pause image build (but allowing general configuration). To say it a different way, we would also not inject security.ima into the rootful pause image.

containers / podman