search

coreos / fedora-coreos-tracker

Issue tracker for Fedora CoreOS
https://fedoraproject.org/coreos/
264 stars 59 forks source link

[rawhide][branched] SELinux AVC denials causes the afterburn write to fail in cloud platforms #1784

Open aaradhak opened 3 months ago

aaradhak commented 3 months ago

Describe the bug

In the recent rawhide & branched cloud platform builds, the kola tests are failing due to an error in the afterburn service.

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: afterburn-sshkeys@core.service; <nil>

On further debugging, it is found that the afterburn process attempted to write to the /var/home/core/.ssh/authorized_keys.d/ directory but was denied by SELinux . This denial caused the afterburn-sshkeys service to fail with a "Permission denied (os error 13)" error.

This seems to be like a selinux-policy issue.

Aug 20 17:00:43.703000 audit[1592]: AVC avc:  denied  { write } for  pid=1592 comm="afterburn" name="authorized_keys.d" dev="nvme0n1p4" ino=20971648 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=0
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]: Error: failed to run
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]: Caused by:
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]:     0: writing ssh keys
Aug 20 17:00:43.705504 afterburn-sshkeys@core.service[1592]:     1: failed to create temporary file
Aug 20 17:00:43.707051 afterburn-sshkeys@core.service[1592]:     2: Permission denied (os error 13) at path "/var/home/core/.ssh/authorized_keys.d/.afterburn-gRJMKD"
Aug 20 17:00:43.707269 init.scope[1]: afterburn-sshkeys@core.service: Main process exited, code=exited, status=1/FAILURE
Aug 20 17:00:43.707481 init.scope[1]: afterburn-sshkeys@core.service: Failed with result 'exit-code'.
Aug 20 17:00:43.707727 init.scope[1]: Failed to start afterburn-sshkeys@core.service - Afterburn (SSH Keys).

Apart from the above AVC denials, came across few other AVC denials in the journal log as below:

Aug 20 17:00:37.025779 kernel: audit: type=1400 audit(1724173234.341:4): avc:  denied  { getattr } for  pid=1367 comm="coreos-boot-mou" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025790 kernel: audit: type=1400 audit(1724173234.357:5): avc:  denied  { read } for  pid=1403 comm="cat" name="bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025801 kernel: audit: type=1400 audit(1724173234.357:6): avc:  denied  { open } for  pid=1403 comm="cat" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025813 systemd[1]: Populated /etc with preset unit settings.
Aug 20 17:00:37.029860 systemd[1]: initrd-switch-root.service: Deactivated successfully.

Reproduction steps

Start a pipeline job build of the kola cloud platforms.

Expected behavior

The SELinux policy to allow the afterburn process to write to the directory in question.

Actual behavior

kola tests fails with this error:

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: afterburn-sshkeys@core.service; <nil>

System details

Kola cloud platform pipeline jobs. Streams - rawhide & branched

Butane or Ignition config

No response

Additional information

There's a similar afterburn issue that was filed against c9s that was fixed, but possibly the fixes there need to be brought to Fedora too: https://issues.redhat.com/browse/RHEL-49735

aaradhak commented 3 months ago

Opened a BZ for this selinux-policy avc denial - https://bugzilla.redhat.com/show_bug.cgi?id=2306352

marmijo commented 2 months ago

A workaround was added for this in: https://github.com/coreos/fedora-coreos-config/pull/3127. We're now able to run the affected kola cloud tests in rawhide and branched.