what does a subject of "exe=/" mean?

[ehousey]

Hi, I recenty installed fapolicyd and started using it. One thing I noticed is that in order to enforce fapolicyd on an EKS worker node, I had to add 3 lib64 libraries (the objects of an allow rule) with a subject of "exe=/" and an object of "path=/usr/lib64/some_lib.

What does a subject of "exe=/" mean? According to the fapolicyd documentation that I referenced (https://www.mankier.com/5/fapolicyd.rules), the subject that is defined by "exe" is "The full path to the executable.". The object with "path=" is defined as "This is the full path to the file that will be accessed."

It is not clear to me what a subject of "exe=/" means.

Is this a bug?

[stevegrubb]

It's not a rule we ship. So, I'd bounce it back to you. What were you trying to do with that rule? Why was it needed? It likely is a no-op since it will never match. Fapolicyd does not validate rules point to anything that exists (so that you can write rules for things that get mounted later). It just checks syntax. fapolicyd-cli on the other hand does some level of validation but it wouldn't see this. fapolicy-analyzer probably would see this as a problem since it does rule linting.

[ehousey]

Hi, thank you for the prompt reply!

I started by running fapolicyd with the "fapolicyd.rules.restrictive" ruleset. The /usr/bin/kubelet binary on the Kubernetes worker node for AWS for whatever reason is not rpm managed by AWS EKS worker nodes, and since it runs pretty much all the time, fapolicyd immediately reported denys with this binary, for example:

rule=13 dec=deny_audit perm=execute auid=-1 pid=22292 exe=/usr/bin/kubelet : path=/opt/cni/bin/aws-cni ftype=application/x-executable trust=0 rule=13 dec=deny_audit perm=execute auid=-1 pid=22293 exe=/usr/bin/kubelet : path=/opt/cni/bin/egress-v4-cni ftype=application/x-executable trust=0 rule=13 dec=deny_audit perm=execute auid=-1 pid=22294 exe=/usr/bin/kubelet : path=/opt/cni/bin/portmap ftype=application/x-executable trust=0 rule=13 dec=deny_audit perm=execute auid=-1 pid=22350 exe=/usr/bin/kubelet : path=/opt/cni/bin/aws-cni ftype=application/x-executable trust=0 rule=13 dec=deny_audit perm=execute auid=-1 pid=22351 exe=/usr/bin/kubelet : path=/opt/cni/bin/egress-v4-cni ftype=application/x-executable trust=0 rule=13 dec=deny_audit perm=execute auid=-1 pid=22352 exe=/usr/bin/kubelet : path=/opt/cni/bin/portmap ftype=application/x-executable trust=0

I then added /usr/bin/kubelet to the fapolicyd.trust file:

This file contains a list of trusted files

#

FULL PATH SIZE SHA256

/usr/bin/kubelet 124788184 cd7e7f1d4c8198deef619d9967e01aed2c592c80441bfc8787403bbad827be61

and restarted fapolicyd with "--debug-deny", at which time fapolicyd would occasionally report these denys:

rule=3 dec=deny_audit perm=open auid=-1 pid=30057 exe=/ : path=/usr/lib64/ libpthread-2.26.so ftype=application/x-sharedlib trust=1 rule=3 dec=deny_audit perm=open auid=-1 pid=30057 exe=/ : path=/usr/lib64/ libpthread-2.26.so ftype=application/x-sharedlib trust=1

(similar denys from libseccomp and libc in addition to libpthread).

If you notice, here is the first time I observed the "exe=/". From these denys, I pieced together these rules, which I put in front of rule 3:

Prevent execution by ld.so

allow perm=open exe=/ : path=/usr/lib64/libpthread-2.26.so ftype=application/x-sharedlib allow perm=open exe=/ : path=/usr/lib64/libseccomp.so.2.4.1 ftype=application/x-sharedlib allow perm=open exe=/ : path=/usr/lib64/libc-2.26.so ftype=application/x-sharedlib deny_audit perm=any pattern=ld_so : all <===== THIS WAS RULE 3 WHICH DENIED THE libpthread ACCESS ABOVE

When I restarted fapolicyd after adding the 3 rules above rule 3, the EKS node started to behave as it should, and I did not observe any more deny messages.

So, maybe I did not put the rules in correctly, but I based my rules on the deny messages and what documentation I could use as a guide.

Did I get the rules correct? I am still not sure what "exe=/" would mean here.

There is another behavior I noticed. While testing the rules above (with the other rules in the restrictive rule set), I noticed that if I ran fapolicyd with "--debug" only, then stopped it after a few minutes, I would see some small number of denies (e.g. "3") in the final report. But capturing the debug output, I could not find any "deny" messages in the output. If I ran the same test with the "--debug-deny" option, the final report would show "0" denys. Both tests are reproducible in that using "--debug" would report some small non-zero number of denys (but they did not show up in the output), and using "--debug-deny" would report "0" denys. Can you make anything of this?

One more question - we are looking to lock down the entire system for FedRAMP compliance. We are concerned that if we just use "rpmdb" and the trust file as the set of trusted files, there is a risk of a malicious RPM getting on the system, and since fapolicyd trusts the RPM database, this may compromise the system. Is it possible to NOT include "rpmdb" in the trusted list, and supply enough rules to allow the system to function? I am about to start experimenting with this approach.

Thanks again, Ed

On Tue, Apr 18, 2023 at 6:42 AM Steve Grubb @.***> wrote:

It's not a rule we ship. So, I'd bounce it back to you. What were you trying to do with that rule? Why was it needed? It likely is a no-op since it will never match. Fapolicyd does not validate rules point to anything that exists (so that you can write rules for things that get mounted later). It just checks syntax. fapolicyd-cli on the other hand does some level of validation but it wouldn't see this. fapolicy-analyzer probably would see this as a problem since it does rule linting.

— Reply to this email directly, view it on GitHub https://github.com/linux-application-whitelisting/fapolicyd/issues/238#issuecomment-1512934101, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATI4XP7RCCOEPTEIRZE5J7DXBZ44BANCNFSM6AAAAAAXBYOGCU . You are receiving this because you authored the thread.Message ID: @.*** com>

[stevegrubb]

The exe=/ is invalid. There cannot be an executable that is the root directory. I have seen issues like this when the overlay file system is in use. It takes a newer version of fapolicyd (1.1.7 or later) to correctly handle overlayfs and bind mounts. But then you may have other problems with containers since fapolicyd has no data source for them.

I don't know why any denies would be missing. Maybe the invalid paths or something that was fixed after the version you are running.

Yes, you can turn off the rpmdb and manage your own trustdb. Just set trust = file. Then add files like this fapolicyd-cli --file add /usr/local/bin. Also, to make managing things better, we have a /etc/fapolicyd/trust.d/ where you can drop off preconfigured trust based on the applications you install or update.

Also, rpm and dnf can validate rpm signatures and not allow installation if the signature check fails. fapolicyd only accepts rpm info that passes rpm's integrity checks.

[ehousey]

Thank you for the detailed response. I will look for the 1.1.7 version of fapolicyd.

I have a few more questions if you don't mind:

if I have this rule in my fapolicyd.rules file as rule #1:

allow_syslog perm=execute exe=/usr/bin/bash : path=/usr/bin/my-ls ftype=application/x-executable

and this line in my fapolicyd.trust file:

/usr/bin/my-ls 109288 b428922e0bd2fce1aa4eb0fb0f4f669c50dcaba309404d2a919765da357c6bc0

and assuming that "/usr/bin/my-ls" exists and that the size and shasum above are correct, what is the difference between the rules entry and the entry in the trust file? I can allow or deny the execution of "/usr/bin/my-ls" by just changing the rule above (allow_syslog vs deny_syslog) and have or not have execution permission of /usr/bin/my-ls, all without putting an entry in fapolicyd.trust.

Secondly, I cannot seem to get the deny_audit or allow_audit to work. I saw some reference to this issue on github I think. Has this issue been resolved?

Lastly, how does "allow_log" and "deny_log" work? Will these settings log the output to some arbitrary log file that we can specify? If so, how do we specify the file? I could not find any documentation on getting these settings to work. I am able to get the output logged to /var/log/messages using deny_syslog and allow_syslog.

Thank you, Ed

On Wed, Apr 19, 2023 at 2:53 PM Steve Grubb @.***> wrote:

The exe=/ is invalid. There cannot be an executable that is the root directory. I have seen issues like this when the overlay file system is in use. It takes a newer version of fapolicyd (1.1.7 or later) to correctly handle overlayfs and bind mounts. But then you may have other problems with containers since fapolicyd has no data source for them.

I don't know why any denies would be missing. Maybe the invalid paths or something that was fixed after the version you are running.

Yes, you can turn off the rpmdb and manage your own trustdb. Just set trust = file. Then add files like this fapolicyd-cli --file add /usr/local/bin. Also, to make managing things better, we have a /etc/fapolicyd/trust.d/ where you can drop off preconfigured trust based on the applications you install or update.

Also, rpm and dnf can validate rpm signatures and not allow installation if the signature check fails. fapolicyd only accepts rpm info that passes rpm's integrity checks.

— Reply to this email directly, view it on GitHub https://github.com/linux-application-whitelisting/fapolicyd/issues/238#issuecomment-1515294243, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATI4XPYN2OLVL5SS7PAOA7TXCA7CFANCNFSM6AAAAAAXBYOGCU . You are receiving this because you authored the thread.Message ID: @.*** com>

[stevegrubb]

That rule does not consider trust. (You can hexedit the file and it will still allow it.) Therefore an entry in the trustdb is not needed. However, without the trust entry, you cannot tell if the file has been replaced with something that is not what you expect. It really just depends on your threat model and whether integrity testing should be done.

If you do want integrity checking, you need to enable what kind of checking such as sha256 or size. Then you need an entry in the trustdb. Then you need to add trust=1 on the object side for the rule.

If you are running in debug mode with it in the foreground, it won't log to the audit logs. Also, the audit system in the kernel has some performance optimization where if there are no audit rules loaded, then it does away with certain classes of audit events. To check it, run auditctl -l. If you don't have any rules, add at least one (maybe watch the shadow file for writes? -F path=/etc/shadow -F perm=w) and restart the audit daemon to make it load rules. This is not a fapolicyd issue, it's how the audit system works.

allow_log/deny_log means send the event to both places for good measure.

[ehousey]

Thank you for the detailed explanations. This all makes good sense!

Ed

On Thu, Apr 20, 2023 at 3:29 PM Steve Grubb @.***> wrote:

That rule does not consider trust. (You can hexedit the file and it will still allow it.) Therefore an entry in the trustdb is not needed. However, without the trust entry, you cannot tell if the file has been replaced with something that is not what you expect. It really just depends on your threat model and whether integrity testing should be done.

If you do want integrity checking, you need to enable what kind of checking such as sha256 or size. Then you need an entry in the trustdb. Then you need to add trust=1 on the object side for the rule.

If you are running in debug mode with it in the foreground, it won't log to the audit logs. Also, the audit system in the kernel has some performance optimization where if there are no audit rules loaded, then it does away with certain classes of audit events. To check it, run auditctl -l. If you don't have any rules, add at least one (maybe watch the shadow file for writes? -F path=/etc/shadow -F perm=w) and restart the audit daemon to make it load rules. This is not a fapolicyd issue, it's how the audit system works.

allow_log/deny_log means send the event to both places for good measure.

— Reply to this email directly, view it on GitHub https://github.com/linux-application-whitelisting/fapolicyd/issues/238#issuecomment-1516906879, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATI4XP6RX4BT4WEIRKM22XTXCGMDLANCNFSM6AAAAAAXBYOGCU . You are receiving this because you authored the thread.Message ID: @.*** com>

[paulrosberg]

Have a similar issue with fapolicyd and a docker install. Get these blocks about 30% of the times I issue a specific command to a container (performing a kill for nginx log rotation):

rule=6 dec=deny_audit perm=open auid=-1 pid=709023 exe=/ : path=/usr/lib64/libseccomp.so.2.5.2 ftype=application/x-sharedlib trust=1 rule=6 dec=deny_audit perm=open auid=-1 pid=709108 exe=/ : path=/usr/lib64/libseccomp.so.2.5.2 ftype=application/x-sharedlib trust=1

Tried abut 10 different rules to allow it with a rule but no go. It seems to totally reject any rules trying to match the above deny. The default rule in 30-patterns.rules catch it everytime. deny_audit perm=any pattern=ld_so : all

So ended up having to add the file /etc/fapolicyd/rules.d/29-docker.rules with this rule that basically whitelist everything for now: allow perm=open all : all

Super ugly but whatever "exe=/" is, it seems to totally confuse fapolicyd.

[stevegrubb]

The problem with any containers is that they typically bind mount onto the file system which causes a change in visibility. (The fanotify API is namespace unaware.) And then depending on how many namespaces they have, there may be nothing in /proc to examine. I have to think there is an alternate way to introspect running containers, but fapolicyd does not support containers at the moment.

[paulrosberg]

but fapolicyd does not support containers at the moment.

Thanks for that piece of information, will remove fapolicyd for machines running containers.

[stevegrubb]

That would be my advice at this point. Containers may be supported in the future. But that would also be dependent on them having a manifest that is detailed enough to reason about their integrity.