ctrl+c'ing the foregrounded fapolicyd ended up blocking everything

[scaarup]

Hi.

I was testing a new rule with fapolicyd, so I was running the daemon in the foreground like: /sbin/fapolicyd --log-denies. The rule seemed to work, so I used ctrl+c to stop the process. After that no one could run anything on the server - and not open a new shell or anything:

Mount change detected
Mount change detected
^Cshutting down...
Inter-thread max queue depth 2
Allowed accesses: 639
Denied accesses: 0
Trust database max pages: 12800
Trust database pages in use: 1560 (12%)
Subject cache size: 2048
Subject slots in use: 108 (5%)
Subject hits: 531
Subject misses: 108
Subject evictions: 0 (0%)
Object cache size: 8192
Object slots in use: 221 (2%)
Object hits: 418
Object misses: 228
Object evictions: 7 (1%)
[root@hostname rules.d]# vim 01-k3s.rules
vim: error while loading shared libraries: libm.so.6: cannot open shared object file: Operation not permitted
[root@hostname rules.d]# vim 01-k3s.rules
vim: error while loading shared libraries: libm.so.6: cannot open shared object file: Operation not permitted
[root@hostname rules.d]# vim 01-k3s.rules
vim: error while loading shared libraries: libm.so.6: cannot open shared object file: Operation not permitted
[root@hostname rules.d]# systemctl stop fapolicyd
-bash: /usr/bin/systemctl: Operation not permitted
[root@hostname rules.d]# vim 01-k3s.rules
vim: error while loading shared libraries: libm.so.6: cannot open shared object file: Operation not permitted
[root@hostname rules.d]# fapolicyd --debug-deny
fapolicyd: error while loading shared libraries: liblua-5.3.so: cannot open shared object file: Operation not permitted
[root@hostname rules.d]# systemctl start fapolicyd
-bash: /usr/bin/systemctl: Operation not permitted
[root@hostname rules.d]#
[root@hostname rules.d]#

The ruleset was and is:

1. allow perm=any uid=0 : trust=0 dir=/data/rancher/k3s/
2. allow perm=open auid=-1 exe=/usr/sbin/xtables-nft-multi : ftype=application/x-sharedlib trust=0
3. allow perm=any all : trust=0 ftype=text/x-python
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,application/java-archive,text/x-java,application/x-java-applet,application/javascript,text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
4. allow perm=any uid=0 : dir=/var/tmp/
5. allow perm=any uid=0 trust=1 : all
6. allow perm=open exe=/usr/bin/rpm : all
7. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
8. deny_audit perm=any pattern=ld_so : all
9. deny_audit perm=any all : ftype=application/x-bad-elf
10. allow perm=open all : ftype=application/x-sharedlib trust=1
11. deny_audit perm=open all : ftype=application/x-sharedlib
12. allow perm=execute all : trust=1
13. allow perm=open all : ftype=%languages trust=1
14. deny_audit perm=any all : ftype=%languages
15. allow perm=any all : ftype=text/x-shellscript
16. deny_audit perm=execute all : all
17. allow perm=open all : all

After a hard reboot, the same rules were loaded again and everything is working fine. This makes me suspect that it was the ctrl+c stopping of the daemon, which made it not properly clean up.... ?

Installed on a Red Hat Enterprise Linux 8.7 version 1.1.3-8.el8_7.1.

Thank you.

[stevegrubb]

By any chance, is allow_filesystem_mark set to 1?

[scaarup]

"allow_filesystem_mark" is not set in the config and not mentioned in "man fapolicyd.conf".

[stevegrubb]

Hmm. The first thing it does after printing "shutting down" is iterate over the list of watches and flush them. After that, it should not be consulted for access decisions. It also prints a warning if it cannot remove a watch. There was a bug fixed in 1.1.4 where you can get 2 instances of fapolicyd running. I don't know if that might be the case, but I don't have much else that might cause the problem since watches are released before any statistics are printed.

[scaarup]

Okay thank you. But even if I were to have started two daemons at the same time, it wouln't cause a complete lockdown, right?

[stevegrubb]

Having 2 active daemons can lockup the system. It uses it's own pid to automatically grant access to it's own requests. But these would be foreign to the other daemon. It's going to be racy and prone to ending badly if one stops while an access request is pending. I have no idea if this is the problem you run into, but it's the only thing we've recently fixed that is somehow related to what you are reporting.

[scaarup]

Having 2 active daemons can lockup the system. It uses it's own pid to automatically grant access to it's own requests. But these would be foreign to the other daemon. It's going to be racy and prone to ending badly if one stops while an access request is pending. I have no idea if this is the problem you run into, but it's the only thing we've recently fixed that is somehow related to what you are reporting.

I was actually able to reproduce the issue. Started an additional daemon in the foreground and system got locked up. I will report this to Red Hat, so they can get this updated. Thank you very much.

[stevegrubb]

That would be bz 2103352. HTH...