Audit username is missing for the new driver, user.loginname=<NA>

[ycaoT]

Describe the bug Field user.loginname is showing as N/A when using modern-bpf

2024-01-02T22:14:53: Alert Interactive root user activity detected (user.loginuid=1002 user.loginname=<NA> user.name=root user.uid=0 proc.cmdline=dockerps file=<NA> container.id=host evt.type=execve evt.res=SUCCESS proc.pid=73960 proc.cwd=/home/ycao/ proc.ppid=73682 proc.pcmdline=bash proc.sid=73680 proc.exepath=/usr/bin/docker group.gid=0 group.name=root container.name=host image=<NA>)

When using eBPF probe: it's able to show though

2024-01-02T22:18:39: Alert Interactive root user activity detected (user.loginuid=1002 user.loginname=ycao user.name=root user.uid=0 proc.cmdline=dockerps file=<NA> container.id=host evt.type=execve evt.res=SUCCESS proc.pid=76937 proc.cwd=/home/ycao/ proc.ppid=76739 proc.pcmdline=bash proc.sid=76737 proc.exepath=/usr/bin/docker group.gid=0 group.name=root container.name=host image=<NA>)

I think falco is pulling the userId and username mappings from /etc/passwd, so it's a regression issue.

How to reproduce it Having a rule with outputs: user.loginname, then follow the doc to run modern BPF and traditional BPF driver in containers: https://falco.org/docs/install-operate/running/#docker-privileged as the following:

For --modern-bpf, you will then see the issue

docker run --name modern-bpf -it -d \
           --privileged \
           -v /var/run/docker.sock:/host/var/run/docker.sock \
           -v /proc:/host/proc:ro \
           falcosecurity/falco-no-driver:latest falco --modern-bpf

For ebpf Probe, it's working as expected

docker run --name old-bpf -it -d \
    --privileged \
    -e FALCO_BPF_PROBE="" \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \
    falcosecurity/falco:latest

Expected behaviour user.loginname shows the correct real user info, like the above outputs on the top.

Screenshots

Environment

• Falco version: latest (0.36.*)

• System info:

• Cloud provider or hardware configuration:

• OS: Amazon Linux 2023

• Kernel:

• Installation method: container

Additional context I also tried to add the following mappings to run for the new modern-bpf, still the user.loginname is not working.

    -v /usr:/host/usr:ro \
    -v /etc:/host/etc:ro \

[incertum]

Hi @ycaoT I have modern BPF running on some prod servers and I cannot confirm the issue, I do see user.loginuid as expected. https://falco.org/docs/reference/rules/supported-fields/#field-class-user

Let's zoom into Amazon Linux 2023, I assume the kernel config CONFIG_AUDIT is set. Could you run Falco and have Falco log all spawned_process and not container and proc.tty!=0 and then ssh into a node and share an anonymized version for that log showing the loginuid etc fields? We can then go from there.

Please feel free to share more details on what activity you aim to monitor and what you expect the loginuid to be. For example user.loginuid should be -1 if there is no audit user. Also in containers the concept of audit users is more unusual.

Lastly, what Falco version are you using?

---

Note that depending on how the audit user is configured on your system Falco may not know about the user name and you'll have to resolve that in your data lake or by other means.

[incertum]

Thanks! Not logging user.loginname w/ Falco as I have to do it off host in our ecosystem, that's why I mentioned that resolving uids to names is not a straight forward task in Linux in many cases. It always depends. Would recommend to first focus on verifying if you get the expected uids.

[ycaoT]

Thanks @incertum, I'll check back again and provide more details. Regarding the last part for user name, if we use the standard linux /etc/passwd for the use ID and user name mappings, I assume that's not currently supported by Falco right? If not, do you think it makes sense to request this as a useful feature?

[incertum]

audit users are different from the regular user ids. Would recommend checking them for separate use case. Ultimately best is always doing hands on experiments. Perhaps log all user.* fields in your experiments to see the differences.

[ycaoT]

@incertum I've modified the issue, sorry it was an issue before the Christmas break, had to recall what's missing from my initial filing. This is a regression issue for the new modern Probe, user.loginname is not working anymore. Just re-tested to verify

[incertum]

ACK, this means you are saying that user.loginuid works as expected for the traditional BPF driver and modern BPF. But user.loginname is working under the same conditions only for the traditional BPF driver BUT NOT for modern BPF?

Did I summarize this correctly?

Mind sharing what you did to test the log outputs?

[ycaoT]

That's correct, please find my sample rule:

- rule: testing_root_rule
  desc: Detects when a user performs activities as root
  condition: spawned_process and not container and proc.tty!=0
  output: Interactive root user activity detected (user.loginuid=%user.loginuid user.loginname=%user.loginname user.name=%user.name user.uid=%user.uid proc.cmdline=%proc.cmdline file=%fd.name container.id=%container.id evt.type=%evt.type evt.res=%evt.res proc.pid=%proc.pid proc.cwd=%proc.cwd proc.ppid=%proc.ppid proc.pcmdline=%proc.pcmdline proc.sid=%proc.sid proc.exepath=%proc.exepath group.gid=%group.gid group.name=%group.name container.name=%container.name image=%container.image.repository)

[incertum]

Still having trouble to replicate the issues. Just a quick test on a dev box with an interactive shell over oh my shell zsh gives both user.loginuid and user.loginname for both drivers (traditional BPF and modern BPF). The puzzling part is that we retrieve only the loginuid kernel side and both drivers then use the same userspace parsing logic.

Hence it must be something much more subtle. Would you be able to share a concrete test setup so that we can replicate the issue?

[ycaoT]

I don't see any special configs I have, one thing I want to confirm is that how did you run the commands? Did you run something like sudo docker logs or did you switch to root user first(sudo su) and then run commands like docker logs? In my case, I did switch to root user first and run commands without sudo, then see outputs like in the description.

[ycaoT]

Never mind, same results.

[ycaoT]

In case anyone wants to help with the issue, you can get a Amazon linux 2023 host through vagrant.

Vagrant.configure("2") do |config|
  config.vm.box = "gbailey/al2023"
  config.vm.hostname = "amazonlinux2023"
  config.vbguest.auto_update = false
  config.vm.synced_folder '.', '/vagrant', disabled: true
end

Once SSH into the VM, create a local falco rule file, falco_rules.local.yaml

- rule: testing_interactive_root_rule 
  desc: Detects when a user performs activities as root 
  condition: spawned_process and not container and proc.tty!=0 
  output: Interactive root user activity detected (user.loginname=%user.loginname user.loginuid=%user.loginuid user.name=%user.name user.uid=%user.uid)
  priority: ALERT

Run the falco agent:

docker run --name modern-bpf -it -d \
           --privileged \
           -v /var/run/docker.sock:/host/var/run/docker.sock \
           -v /proc:/host/proc:ro \
           -v ./falco_rules.local.yaml:/etc/falco/falco_rules.local.yaml \
           falcosecurity/falco-no-driver:latest falco --modern-bpf

You should be able to see the outputs from docker logs

[incertum]

It's not OS specific nor an actual Falco issue. I ran the container (however using falcosecurity/falco-no-driver:master) and can see the missing login usernames as well. But when I run the compiled Falco binary from master on my box natively the login usernames are there (same condition and test).

Hence I am deducting that it's an issue with some container mounts or other settings. Not the expert on the Falco upstream containers. Tagging @LucaGuerra @FedeDP @Andreagit97 and @jasondellaluce for help with investigating this. Please note some maintainers are still out until next week.

[incertum]

Actually @deepskyblue86 aren't you our container users and mounts experts 🙃 thanks in advance Angelo!

[ycaoT]

Much thanks @incertum for getting to the bottom of the issue.

While you all are here, I want to ask if any of you can give some code pointers for how falco fetches audit user info. The reason why I am asking is that, currently the audit user info is not working if commands/syscalls are inside containers; but I think it has a legit reason/is possible to implement, such as for system audit purpose.

I am interested to help implement such if I could get a bit more info on how this works for hosts.

[incertum]

@deepskyblue86 worked most on it lately, I never touched the user names logic but it all lives in https://github.com/falcosecurity/libs userspace. Let's wait for Angelo to provide us with some pointers.

[deepskyblue86]

What I did was 100% userspace. Nothing comes up in my mind about why with modern bpf we have such an issue... :thinking: I can just confirm that /etc:/host/etc:ro is required (https://github.com/falcosecurity/libs/blob/master/userspace/libsinsp/user.cpp#L60)

[incertum]

In https://github.com/falcosecurity/falco/issues/2990#issuecomment-1876267628 I don't see /etc:/host/etc:ro (I also forgot to cross-reference with our docs actually), but @ycaoT you also mentioned earlier at some point that it was working with ebpf but not modern_ebpf driver? Mind double checking?

[ycaoT]

@incertum /etc:/host/etc:ro is not on the doc here, https://falco.org/docs/install-operate/running/#modern-ebpf. Checking again

[incertum]

@incertum /etc:/host/etc:ro is not on the doc here, https://falco.org/docs/install-operate/running/#modern-ebpf. Checking again

oh oh, would you mind opening a PR against https://github.com/falcosecurity/falco-website to update the docs?

[ycaoT]

Thanks @deepskyblue86 @incertum . By adding that, I can see it shows, but not all events.

2024-01-09T17:55:18.739020340+0000: Alert Interactive root user activity detected (user.loginname=vagrant user.loginuid=1000 user.name=root user.uid=0)
2024-01-09T17:55:20.904787060+0000: Alert Interactive root user activity detected (user.loginname=vagrant user.loginuid=1000 user.name=root user.uid=0)
2024-01-09T17:55:27.065346515+0000: Alert Interactive root user activity detected (user.loginname=<NA> user.loginuid=1000 user.name=root user.uid=0)
2024-01-09T17:55:27.067583692+0000: Alert Interactive root user activity detected (user.loginname=<NA> user.loginuid=1000 user.name=root user.uid=0)

[incertum]

Awesome, re it not showing in all events, this may be trickier to debug and won't land for Falco 0.37.0 but we will keep checking and attempt to improve things!

[ycaoT]

Closing this, made the PR to update that on falco docs. https://github.com/falcosecurity/falco-website/pull/1236. Thank you the community