search

linux-application-whitelisting / fapolicyd

File Access Policy Daemon
GNU General Public License v3.0
192 stars 55 forks source link

fapolicyd blocks in permissive mode #216

Open cutrightjm opened 1 year ago

cutrightjm commented 1 year ago

Under certain circumstances, I am getting an fapolicyd error where fapolicyd is seemingly ignoring all rules in rules.d/ and blocking. When this occurs, any time that a rule should be hit to permit, the error "Error receiving fanotify_event (Permission denied)"

I am currently unsure what is causing the issue; a reboot has fixed it on one machine, but I ended up reimaging another rather than taking the time to figure it out. Troubleshooting so far has not been successful. Red Hat has a snippet In rare cases, removing the fapolicyd pipe file can solve a lockup, but doesn't mention what those rare cases look like. Running the mentioned command does not seem to fix the issue.

It seems that the relevant code for the error seems to be on line 374 of notify.c: https://github.com/linux-application-whitelisting/fapolicyd/blob/main/src/daemon/notify.c#L374.

I am more than happy to produce any troubleshooting data or rules files when this issue pops up again. Currently these systems are running RHEL 8.7 and fully STIGed, minus fapolicyd in permissive mode and SELinux user contexts.

At the least, when this error is hit, is it possible to recognize that fapolicyd is in permissive mode and fail open?

radosroka commented 1 year ago

Hello,

Are there any AVC?

At the least, when this error is hit, is it possible to recognize that fapolicyd is in permissive mode and fail open?

Not really. But I would say it does not matter in this case.

cutrightjm commented 1 year ago

I have yet to see any denies from SELinux when this error occurs, but am more than happy to double check when the I can get the issue to occur again.

stevegrubb commented 1 year ago

All kernel checks that return EPERM in the fanotify code require CAP_SYS_ADMIN...which we leave on. This is strange because poll has just told us there is something to read and then we go through 4 if statements and read. Read is not normally the enforcement point of policy, opening the descriptor is where that is done. I can't see how that errno is returned.

If you change to an 8.6 or 8.5 kernel, does it also lockup? And you are saying that has been see on multiple machines?

cutrightjm commented 1 year ago

The servers I am mainly having issues with are all running Axway SecureTransport, but the policy listed below is currently in development, and is on around 30 servers. I have not been able to clone the VM to downgrade kernels yet; I can try that soon.

I'm not 100% sure what is causing the issue to trigger, since this same policy is on two other servers running SecureTransport. If I do swap kernels and can't replicate the issue, I will be unsure whether or not it is because I am doing something incorrectly to reproduce, or if that actually solves the problem.

I cannot remember for certain whether or not this issue has occurred on other hosts, but I don't believe it has. I will check with my coworkers tomorrow.

Server info (latest kernel is the one running)

[root@server ~]# rpm -qa | grep kernel
kernel-4.18.0-372.9.1.el8.x86_64
kernel-4.18.0-372.19.1.el8_6.x86_64
kernel-4.18.0-425.3.1.el8.x86_64
kernel-modules-4.18.0-372.9.1.el8.x86_64
kernel-modules-4.18.0-372.19.1.el8_6.x86_64
kernel-modules-4.18.0-425.3.1.el8.x86_64
kernel-tools-libs-4.18.0-425.3.1.el8.x86_64
kernel-tools-4.18.0-425.3.1.el8.x86_64
kernel-core-4.18.0-372.9.1.el8.x86_64
kernel-core-4.18.0-372.19.1.el8_6.x86_64
kernel-core-4.18.0-425.3.1.el8.x86_64

[root@server ~]# rpm -qa | grep fapolicyd
rpm-plugin-fapolicyd-4.14.3-24.el8_7.x86_64
fapolicyd-selinux-1.1.3-8.el8.noarch
fapolicyd-1.1.3-8.el8.x86_64

Below is an example trigger from Ansible, since it's easily reproducable:

[root@ansible scratchpad]# ansible server -m ping
server | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/libexec/platform-python"
    },
    "changed": false,
    "ping": "pong"
}
[root@ansible scratchpad]# ansible server -m ping
[WARNING]: sftp transfer mechanism failed on [server]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: scp transfer mechanism failed on [server]. Use ANSIBLE_DEBUG=1 to see detailed information
[WARNING]: piped transfer mechanism failed on [server]. Use ANSIBLE_DEBUG=1 to see detailed information
server | FAILED! => {
    "msg": "failed to transfer file to /root/.ansible/tmp/ansible-local-600244o6bkty_1/tmpx9kfnm2_ /home/adm_home/ansible/.ansible/tmp/ansible-tmp-1671647093.810531-600247-130439014020343/AnsiballZ_ping.py:\n\ndd: failed to open '/home/adm_home/ansible/.ansible/tmp/ansible-tmp-1671647093.810531-600247-130439014020343/AnsiballZ_ping.py': Operation not permitted\n"
}
[root@ansible scratchpad]#

Here is what that failure looks like on the client with issues

[root@server ~]# systemctl status fapolicyd
● fapolicyd.service - File Access Policy Daemon
   Loaded: loaded (/usr/lib/systemd/system/fapolicyd.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-12-21 12:52:52 EST; 32s ago
     Docs: man:fapolicyd(8)
  Process: 121645 ExecStart=/usr/sbin/fapolicyd (code=exited, status=0/SUCCESS)
  Process: 121609 ExecStartPre=/usr/sbin/fagenrules (code=exited, status=0/SUCCESS)
 Main PID: 121646 (fapolicyd)
    Tasks: 4 (limit: 204257)
   Memory: 43.6M
   CGroup: /system.slice/fapolicyd.service
           └─121646 /usr/sbin/fapolicyd

Dec 21 12:52:53 server  fapolicyd[121646]: Checking if the trust database up to date
Dec 21 12:52:53 server  fapolicyd[121646]: Importing trust data from rpmdb backend
Dec 21 12:52:53 server  fapolicyd[121646]: Importing trust data from file backend
Dec 21 12:52:53 server  fapolicyd[121646]: Entries in trust DB: 30072
Dec 21 12:52:53 server  fapolicyd[121646]: Loaded trust info from all backends(without duplicates): 30072
Dec 21 12:52:53 server  fapolicyd[121646]: Trust database checks OK
Dec 21 12:52:53 server  fapolicyd[121646]: Starting to listen for events
Dec 21 12:53:21 server  fapolicyd[121646]: Error receiving fanotify_event (Permission denied)
Dec 21 12:53:21 server  fapolicyd[121646]: Error receiving fanotify_event (Permission denied)
Dec 21 12:53:21 server  fapolicyd[121646]: Error receiving fanotify_event (Permission denied)

Current fapolicyd configuration

[root@server ~]# cat /etc/fapolicyd/fapolicyd.conf
#
# This file controls the configuration of the file access policy daemon.
# See the fapolicyd.conf man page for explanation.
#

permissive = 1
nice_val = 14
q_size = 640
uid = fapolicyd
gid = fapolicyd
do_stat_report = 1
detailed_report = 1
db_max_size = 40
subj_cache_size = 1549
obj_cache_size = 8191
watch_fs = ext2,ext3,ext4,tmpfs,xfs,vfat,iso9660
trust = rpmdb,file
integrity = none
syslog_format = rule,dec,perm,auid,pid,exe,:,path,ftype,trust
rpm_sha256_only = 0

Current policy

[root@server fapolicyd]# fapolicyd-cli --list
1. allow perm=any exe=/opt/sc/support/bin/php : path=/usr/lib64/
2. allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/jre/
3. allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/lib/
4. allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/bin/
5. allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/mariadb/
6. allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/tomcat/
7. allow perm=any dir=/opt/McAfee/ : dir=/opt/McAfee/
8. allow perm=any exe=nfsd : dir=/nfs/
-> %languages=application/x-bytecode.ocaml,application/x-bytecode.python,application/java-archive,text/x-java,application/x-java-applet,application/javascript,text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
9. allow perm=any uid=0 : dir=/var/tmp/
10. allow perm=any uid=0 trust=1 : all
11. allow perm=open exe=/usr/bin/rpm : all
12. allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
13. deny_log perm=any pattern=ld_so : all
14. allow perm=open dir=/usr/lib/jvm/ : dir=/var/lib/pki/pki-tomcat/work/Catalina/localhost/
15. allow perm=open dir=/usr/lib/jvm/ : dir=/usr/share/tomcat/bin/ ftype=application/java-archive
16. deny_log perm=any all : ftype=application/x-bad-elf
17. allow perm=open all : ftype=application/x-sharedlib trust=1
18. deny_log perm=open all : ftype=application/x-sharedlib
19. allow perm=execute all : trust=1
20. allow perm=any exe=/usr/libexec/platform-python3.6 uid=ansible : dir=/home/ansible
21. allow perm=any exe=/usr/libexec/platform-python3.6 uid=ansible : dir=/home/adm_home/ansible
22. allow perm=any exe=/usr/libexec/openssh/sftp-server uid=ansible : dir=/home/ansible
23. allow perm=any exe=/usr/bin/scp uid=ansible : dir=/home/ansible
24. allow perm=any exe=/usr/libexec/openssh/sftp-server uid=ansible : dir=/home/adm_/home/ansible
25. allow perm=any exe=/usr/bin/scp uid=ansible : dir=/home/adm_home/ansible
26. allow perm=open all : ftype=%languages trust=1
27. deny_log perm=any all : ftype=%languages
28. allow perm=any all : ftype=text/x-shellscript
29. deny_log perm=execute all : all
30. allow perm=open all : all
31. deny_log perm=any all : all
32. deny perm=any all : all

An additional fapolicyd output in debug mode:

[root@server fapolicyd]# fapolicyd --debug-deny
Loading rule file:
## This file is automatically generated from /etc/fapolicyd/rules.d
allow perm=any exe=/opt/sc/support/bin/php : path=/usr/lib64/
allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/jre/
allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/lib/
allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/bin/
allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/mariadb/
allow perm=any exe=/opt/TMWD/SecureTransport/jre/bin/java : dir=/opt/TMWD/SecureTransport/tomcat/
allow perm=any dir=/opt/McAfee/ : dir=/opt/McAfee/
allow perm=any exe=nfsd : dir=/nfs/
%languages=application/x-bytecode.ocaml,application/x-bytecode.python,application/java-archive,text/x-java,application/x-java-applet,application/javascript,text/javascript,text/x-awk,text/x-gawk,text/x-lisp,application/x-elc,text/x-lua,text/x-m4,text/x-nftables,text/x-perl,text/x-php,text/x-python,text/x-R,text/x-ruby,text/x-script.guile,text/x-tcl,text/x-luatex,text/x-systemtap
allow perm=any uid=0 : dir=/var/tmp/
allow perm=any uid=0 trust=1 : all
allow perm=open exe=/usr/bin/rpm : all
allow perm=open exe=/usr/libexec/platform-python3.6 comm=dnf : all
deny_log perm=any pattern=ld_so : all
allow perm=open dir=/usr/lib/jvm/ : dir=/var/lib/pki/pki-tomcat/work/Catalina/localhost/
allow perm=open dir=/usr/lib/jvm/ : dir=/usr/share/tomcat/bin/ ftype=application/java-archive
deny_log perm=any all : ftype=application/x-bad-elf
allow perm=open all : ftype=application/x-sharedlib trust=1
deny_log perm=open all : ftype=application/x-sharedlib
allow perm=execute all : trust=1
allow perm=any exe=/usr/libexec/platform-python3.6 uid=ansible : dir=/home/ansible
allow perm=any exe=/usr/libexec/platform-python3.6 uid=ansible : dir=/home/adm_home/ansible
allow perm=any exe=/usr/libexec/openssh/sftp-server uid=ansible : dir=/home/ansible
allow perm=any exe=/usr/bin/scp uid=ansible : dir=/home/ansible
allow perm=any exe=/usr/libexec/openssh/sftp-server uid=ansible : dir=/home/adm_/home/ansible
allow perm=any exe=/usr/bin/scp uid=ansible : dir=/home/adm_home/ansible
allow perm=open all : ftype=%languages trust=1
deny_log perm=any all : ftype=%languages
allow perm=any all : ftype=text/x-shellscript
deny_log perm=execute all : all
allow perm=open all : all
deny_log perm=any all : all
deny perm=any all : all
Loaded 32 rules
Changed to uid 993
Initializing the trust database
fapolicyd integrity is 0
backend rpmdb registered
backend file registered
Loading rpmdb backend
Loading file backend
Checking if the trust database up to date
Importing trust data from rpmdb backend
Importing trust data from file backend
Entries in trust DB: 30072
Loaded trust info from all backends(without duplicates): 30072
Trust database checks OK
added /dev/shm mount point
added / mount point
added /boot mount point
added /boot/efi mount point
added /var mount point
added /var/tmp mount point
added /home mount point
added /opt mount point
added /tmp mount point
added /var/log mount point
added /var/log/audit mount point
added /run/user/992 mount point
added /run/user/768200033 mount point
added /run/user/768200009 mount point
Starting to listen for events
## Running ansible ping from ansible host
Mount change detected
Added /run/user/768200028 mount point
Error receiving fanotify_event (Permission denied)
Error receiving fanotify_event (Permission denied)
Error receiving fanotify_event (Permission denied)

Searching for any AVC messages

[root@server ~]# ausearch -m avc -ts today
<no matches>
stevegrubb commented 1 year ago

Does Axway SecureTransport have any kernel modules? This problem is originating in the kernel. It won't matter what policy or configuration fapolicyd has. The read syscall is failing with EPERM. If there are 3rd party kernel modules being loaded, I'd suggest unloading them and then trying fapolicyd again.

stevegrubb commented 1 year ago

Also, there is no way to tell the kernel to allow access because the read that is failing has the metadata that we need to use to grant access.

cutrightjm commented 1 year ago

Hello,

I don't think Axway ST has any of its own kernel modules, but it does have a few additional that I've noticed so far over our baseline. These servers are serving and working with files (mainly .txts) on a mounted NFS share, if that helps:

dns_resolver
fscache
grace
ib_core
inet_diag
lockd
mptcp_diag
nf_reject_ipv4
nf_reject_ipv6
nfs
nfsv4
nf_tables_set
nft_ct
nft_fib
nft_fib_inet
nft_fib_ipv4
nft_fib_ipv6
nft_reject
nft_reject_inet
raw_diag
rpcsec_gss_krb5
tcp_diag
udp_diag
unix_diag
vsock_diag

There is a 3rd Axway ST server that is not using NFS shares. So far, we have not experienced the issue on that server.