Closed manojdeshmukh45 closed 5 months ago
Andreagit97
commented
1 year ago thank you for reporting this! do you have a repro to trigger this rule with all <NA> fields?
Please note that since you are on the host all containers and k8s data are <NA> and this is fine.
cristiklein
commented
1 year ago Hello,
We bumped into this issue quite a few times and decided to make a deep-dive.
As an example, we get the following log lines from some staging Kubernetes cluster:
09:06:37.105020032: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=sh proc_exepath=/bin/sh parent=xargs command=sh -c redis-cli -p 6379 info replication \| grep master_sync_in_progress:1 \| tr -d "\r" \| tr -d "\n" terminal=0 exe_flags=<NA> k8s.ns=redis-system k8s.pod=rfr-redis-cluster-2 container=73897ca9fa41)
09:06:37.105848477: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=tr proc_exepath=/usr/bin/tr parent=sh command=tr -d \n terminal=0 exe_flags=<NA> k8s.ns=redis-system k8s.pod=rfr-redis-cluster-2 container=73897ca9fa41)What's really weird is that it almost looks like all coreutils trigger this event. :smile:
We looked into the container image and the source of the events seems to be the fact that busybox does setuid, even if not needed. (See busybox code.)
This can also be verified as follows:
Spin up an Ubuntu 22.04.3 LTS machine. Your regular development laptop works just fine.
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"Make sure you have Docker working:
$ docker --version
Docker version 24.0.5, build ced0996Run the setuids.bt tool from bpftrace:
$ docker run -ti -v /usr/src:/usr/src:ro -v /lib/modules/:/lib/modules:ro -v /sys/kernel/debug/:/sys/kernel/debug:rw --net=host --pid=host --privileged quay.io/iovisor/bpftrace:v0.18.1 setuids.bt
Attaching 7 probes...
Tracing setuid(2) family syscalls. Hit Ctrl-C to end.
TIME PID COMM UID SYSCALL ARGS (RET)
10:06:57 3023954 ping 1000 setuid uid=1000 (0)
[...]Ignore all the setuid's happening on your system. :smile:
While you're at it, install Falco as describe here. Pick Modern eBPF driver. You can check that Falco is running as follows:
$ journalctl --unit falco-modern-bpf.service
sep 01 11:50:15 minitruee systemd[1]: Started Falco: Container Native Runtime Security with modern e>
sep 01 11:50:16 minitruee falco[3007632]: Falco version: 0.35.1 (x86_64)
sep 01 11:50:16 minitruee falco[3007632]: Falco initialized with configuration file: /etc/falco/falc>
sep 01 11:50:16 minitruee falco[3007632]: Loading rules from file /etc/falco/falco_rules.yaml
sep 01 11:50:16 minitruee falco[3007632]: Loading rules from file /etc/falco/falco_rules.local.yaml
sep 01 11:50:16 minitruee falco[3007632]: The chosen syscall buffer dimension is: 8388608 bytes (8 M>
sep 01 11:50:16 minitruee falco[3007632]: Starting health webserver with threadiness 8, listening on>
sep 01 11:50:16 minitruee falco[3007632]: Loaded event sources: syscall
sep 01 11:50:16 minitruee falco[3007632]: Enabled event sources: syscall
sep 01 11:50:16 minitruee falco[3007632]: Opening 'syscall' source with modern BPF probe.
sep 01 11:50:16 minitruee falco[3007632]: One ring buffer every '2' CPUs.
[...]In another terminal, run a busybox image:
$ docker run --rm --user 1000 busybox cat
[exists immediately]Observe the output of setuids.bt:
10:08:13 3026177 cat 1000 setuid uid=1000 (0)Mkey, so I guess busybox likes to setuid to itself. Doesn't feel like a big security threat. What does Falco say?
I tried both the Kmod and Modern eBPF driver. The busybox command above generates the following warning:
sep 01 12:08:13 minitruee falco[3007632]: 12:08:13.458396596: Notice Unexpected setuid call by non-sudo, non-root program (user=<NA> user_loginuid=-1 cur_uid=1000 parent=<NA> command=cat pid=3026177 uid=<NA> container_id=3401b97a71fd image=<NA>)Judging by the timestamp, both Falco and setuids.bt report the same event (my laptop is in UTC+2, hence the 2 hours discrepancy).
However, Falco does not seem to report args.uid correctly. I tried both the Modern eBPF and the Kmod driver, and they both behave the same. If Falco reported args.uid correctly, it would be trivial to ignore events like args.uid == cur_uid.
Any clues where to dig further on why Falco reports args.uid incorrectly?
Andreagit97
commented
1 year ago @cristiklein thank you very much for the very detailed investigation!
Mkey, so I guess busybox likes to setuid to itself. Doesn't feel like a big security threat. What does Falco say?
Yeah it seems weird, we need to investigate more on this, thanks for the pointer!
However, Falco does not seem to report args.uid correctly. I tried both the Modern eBPF and the Kmod driver, and they both behave the same. If Falco reported args.uid correctly, it would be trivial to ignore events like args.uid == cur_uid. Any clues where to dig further on why Falco reports args.uid incorrectly?
Thanks to your repro probably we will discover why, it could take some time since we are in the middle of a release but for sure is something we want to investigate! Thank you again for all the precious data!
cristiklein
commented
1 year ago Thanks @Andreagit97 for your quick response.
I discovered one more thing that will hopefully help narrow down the culprit.
If I change the rule's output and add:
args=%evt.args uid_via_index=%evt.arg[0] info=%evt.info uid_via_rawargs=%evt.rawarg.uid
Then Falco's output becomes:
14:30:28.999973619: Notice Unexpected setuid call by non-sudo, non-root program (user=
user_loginuid=-1 cur_uid=1000 parent= command=cat pid=3085201 uid= container_id=d9f922a096f2 image= args=uid=1000( ) uid_via_index= info=uid=1000( ) uid_via_rawargs= )
Interestingly, it seems like uid=1000 is correctly captured in evt.args and evt.info, but is somehow not transferred into evt.arg. :shrug:
Good luck with the release! Looking forward to it.
manojbamboobox
commented
1 year ago Though I added the above to output, i'm getting blank.
cristiklein
commented
1 year ago Though I added the above to output, i'm getting blank.
What do you mean? I can clearly see uid=<NA> and info=uid=0 in your screenshot above. Seems like your offending process is running with uid 0, which is available to Falco in the lower layers, but not properly propagated into the event args.
manojdeshmukh45
commented
1 year ago What to do for this issue. Im getting 0.
On Wed, 18 Oct, 2023, 4:49 pm Cristian Klein, @.***> wrote:
Though I added the above to output, i'm getting blank. [image: image] https://user-images.githubusercontent.com/138666148/276120551-f25152c3-eb09-4173-84f1-0806ca985d20.png
What do you mean? I can clearly see uid=
and info=uid=0 in your screenshot above. Seems like your offending process is running with uid 0, which is available to Falco in the lower layers, but not properly propagated into the event args. — Reply to this email directly, view it on GitHub https://github.com/falcosecurity/libs/issues/1630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIRWHSEKLC5T466UAT5UL53X763KZAVCNFSM6AAAAAA3CNKZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRYGIZTMOJRGE . You are receiving this because you authored the thread.Message ID: @.***>
cristiklein
commented
1 year ago What version of Falco are you using? (I used Falco 0.35.1.)
manojdeshmukh45
commented
1 year ago {"default_driver_version":"6.0.1+driver","driver_api_version":"5.0.0","driver_schema_version":"2.0.0","engine_version":"26","falco_version":"0.36.0","libs_version":"0.13.1","plugin_api_version":"3.1.0"}
On Wed, Oct 18, 2023 at 7:08 PM Cristian Klein @.***> wrote:
What version of Falco are you using? (I used Falco 0.35.1.)
— Reply to this email directly, view it on GitHub https://github.com/falcosecurity/libs/issues/1630, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIRWHSE7WC55JTSATOMPDGDX77LVZAVCNFSM6AAAAAA3CNKZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRYGQ4DCMZSGE . You are receiving this because you authored the thread.Message ID: @.***>
--
Thanks & Regards
Manoj Deshmukh
Cybersecurity Analyst
CEH Practical v12 | CompTIA CySA+
cristiklein
commented
1 year ago I re-ran my test with Falco 0.36.1. In the meantime, the "Unexpected setuid call" rule was moved to falco-incubating_rules.yaml, so I had to do the following first:
$ sudo wget https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml -O /etc/falco/rules.d/falco-incubating_rules.yaml
# Falco 0.36.1 seem to not like the `required_engine_version` key.
$ sudo sed -i -e 's@- required_engine_version@#- required_engine_version@' /etc/falco/rules.d/falco-incubating_rules.yaml
# Also print %evt.args
sudo sed -i -e 's@output: Unexpected setuid.*$@& args=%evt.args@' /etc/falco/rules.d/falco-incubating_rules.yamlThen I ran a busybox image:
$ docker run --rm --user 1000 busybox cat
[exits immediately]Falco output the following:
okt 19 08:54:39 minitruee falco[1211656]: 08:54:39.141114250: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=cat proc_exepath=/bin/cat parent=containerd-shim command=cat terminal=0 exe_flags=<NA> container_id=556312d7d5d0 container_name=<NA>) args=uid=1000(<NA>)As you can see %evt.args contains the correct uid, but `%evt.arg.uid% does not.
So it seems like there is a bug in the logic that populates %evt.arg.
What do you think?
FedeDP
commented
1 year ago @cristiklein can you try using evt.rawarg.uid instead?
FedeDP
commented
1 year ago So, i just tried to repro this with sinsp-example from libs master, that is a small executable on top of libsinsp, and it seems i receive events correctly:
sudo ./libsinsp/examples/sinsp-example -m -f "evt.type=setuid and evt.dir=>" -o "%evt.type %evt.args arg=%evt.arg.uid user:%user.name raw=%evt.rawarg.uid"
setuid uid=0(root) arg=root user:root raw=
setuid uid=1000(federico) arg=federico user:root raw=
setuid uid=1000(federico) arg=federico user:federico raw=This was captured while doing in another terminal: docker run --rm --user 1000 busybox cat
Notes:
rawarg is empty :/rawarg before anything else, it won't print anything after it: sudo ./libsinsp/examples/sinsp-example -m -f "evt.type=setuid and evt.dir=>" -o "raw=%evt.rawarg.uid %evt.type %evt.args arg=%evt.arg.uid user:%user.name"
raw=
raw=
raw=/cc @jasondellaluce for these 2 issues, perhaps you got some ideas?
cristiklein
commented
1 year ago @cristiklein can you try using
evt.rawarg.uidinstead?
Seems like evt.rawarg.uid is also empty:
cklein@minitruee:~$ sudo wget https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml -O /etc/falco/rules.d/falco-incubating_rules.yaml
--2023-10-19 17:27:48-- https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 64418 (63K) [text/plain]
Saving to: ‘/etc/falco/rules.d/falco-incubating_rules.yaml’
/etc/falco/rules.d/falco- 100%[==================================>] 62,91K --.-KB/s in 0,01s
2023-10-19 17:27:48 (5,57 MB/s) - ‘/etc/falco/rules.d/falco-incubating_rules.yaml’ saved [64418/64418]
cklein@minitruee:~$ sudo sed -i -e 's@- required_engine_version@#- required_engine_version@' /etc/falco/rules.d/falco-incubating_rules.yaml
cklein@minitruee:~$ sudo sed -i -e 's@output: Unexpected setuid.*$@& args=%evt.args rawarg_uid=%evt.rawarg.uid@' /etc/falco/rules.d/falco-incubating_rules.yaml
cklein@minitruee:~$ cat /etc/falco/rules.d/falco-incubating_rules.yaml | grep rawarg
output: Unexpected setuid call by non-sudo, non-root program (arg_uid=%evt.arg.uid evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info) args=%evt.args rawarg_uid=%evt.rawarg.uid
cklein@minitruee:~$ docker run --rm --user 1000 busybox sleep 10And the output of Falco becomes:
okt 19 17:30:04 minitruee falco[1286358]: 17:30:04.278000854: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=sleep proc_exepath=/bin/sleep parent=containerd-shim command=sleep 10 terminal=0 exe_flags=<NA> container_id=4ed258f5a0fa container_name=<NA>) args=uid=1000(<NA>) rawarg_uid=<NA>https://github.com/falcosecurity/libs/pull/1428 will fix the rawarg issue ;)
Also, about:
As you can see %evt.args contains the correct uid, but `%evt.arg.uid% does not.
%evt.arg.uid is resolved to the user name matching the uid, that in this case is "NA":
okt 19 08:54:39 minitruee falco[1211656]: 08:54:39.141114250: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=
evt_type=setuid user= user_uid=1000 user_loginuid=-1 process=cat proc_exepath=/bin/cat parent=containerd-shim command=cat terminal=0 exe_flags= container_id=556312d7d5d0 container_name= ) args=uid=1000( )
Instead, rawarg.uid should print the number without further conversion. But rawarg was broken for non-strings params :)
Hopefully my patch make it to a Falco 0.36.2 patch release!
poiana
commented
9 months ago Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://github.com/falcosecurity/community.
/lifecycle stale
Andreagit97
commented
9 months ago /remove-lifecycle stale
Andreagit97
commented
9 months ago it's on my plate, I'm investigating between one thing and another, and hope to come up with a solution soon!
Andreagit97
commented
9 months ago uhm ok, the issue should be clear now, we need to understand how to tackle it!
Running docker run --rm --user 1000 busybox cat means that you are running the processes inside the container as the user 1000 on the host. What we do in the code, and please correct me if I'm wrong @FedeDP is to try to search the user 1000 inside the container but of course, we won't find it because the user 1000 is the one outside the container, and for this reason we cannot resolve the name...
evt.arg.uid returns <NA> because it tries to resolve the username under the hood, so it correctly contains 1000 but before returning it to Falco we try to convert it to the user_name, we don't find it and we return <NA>. The reason why evt.rawarg.uid return 1000 is that for this field we don't perform the resolution
Andreagit97
commented
9 months ago I will move it to https://github.com/falcosecurity/libs since this seems a libsinsp issue
Andreagit97
commented
9 months ago How to reproduce it with sinsp-example
sudo ./libsinsp/examples/sinsp-example \
-o "user_id=%evt.rawarg.uid user_name=%evt.arg.uid" \
-f "evt.type in (setuid) and evt.dir=> and container.id!=host" --modern_bpfdocker run --rm --user 1000 busybox catuser_id=1000 user_name=<NA>
user_id=1000 user_name=<NA>Yep, IMHO this is not a bug: if user 1000 is not existing in the container, it is fine to return NA username, right? At the same time, if you enforce an existing (in the container) user, everything works fine.
Edit: we might add a fallback lookup for the uid/gid on the host when the container has no such uid/gid perhaps. I am not sure whether that makes sense though.
Andreagit97
commented
9 months ago Yep, IMHO this is not a bug: if user 1000 is not existing in the container, it is fine to return NA username, right?
Uhm IMO Falco should try to give the right user since it runs on the host and has the possibility to see all the users...
Edit: we might add a fallback lookup for the uid/gid on the host when the container has no such uid/gid perhaps.
This could be an idea, I am not sure if there are other cases when the user is not defined inside the container but it doesn't correspond to any user on the host, I don't think so :thinking: So the fallback might be ok! @deepskyblue86 WDYT?
FedeDP
commented
9 months ago But please note:
docker run --rm --user 1000 busybox whoami
whoami: unknown uid 1000I still think NA is the proper answer here.
Andreagit97
commented
9 months ago yeah, docker knows nothing about user 1000 because is a user outside the container, but Falco as an external tool has the knowledge to understand who is the real user...the downside of leaving all as it is is that all the processes inside the containers will have user.name = <NA> :/
FedeDP
commented
9 months ago docker knows nothing about user 1000 because is a user outside the container,
I am not sure what --user does, but i guess it just sets setuid for the process inside the container;
see:
docker run --rm --user federico busybox whoami
docker: Error response from daemon: unable to find user federico: no matching entries in passwd file.the downside of leaving all as it is is that all the processes inside the containers will have user.name =
It's not a downside, it is exactly what is happening. Saying that, for example, docker run --rm --user 1000 busybox cat runs as federico inside the container is simply not true.
Found this on stackoverflow:
Is it safe to use $ docker run -u or USER variable in Dockerfile with a nonexistent user? Yes. The two things that are important are (a) whether or not the numeric user ID is 0 and (b) whether the numeric user ID matches the ownership of files in the container or mounted volumes. If the numeric user ID doesn't match then the user won't be able to overwrite files. ... There's no requirement to "create the user" before you do, and it's not an especially good practice to build an image that has a specific host user ID built in.
FedeDP
commented
6 months ago /milestone 0.17.0
FedeDP
commented
5 months ago Is anybody still experiencing issues on Falco 0.37? Moving to milestone 0.18.0, but ready to close this one if the issue is gone. /milestone 0.18.0
Andreagit97
commented
5 months ago The decision of closing or not this is on us... we identified the issue, if we decide that we are fine with the <NA> since the container cannot see the external user, i think we can close it
FedeDP
commented
5 months ago I think so, as i explained earlier it would be wrong to state the opposite since if the user is not existent in the container, it is to all effects NA to it.
/milestone 0.17.0
/close
poiana
commented
5 months ago @FedeDP: The provided milestone is not valid for this repository. Milestones in this repository: [0.18.0, TBD, next-driver]
Use /milestone clear to clear the milestone.
poiana
commented
5 months ago @FedeDP: Closing this issue.
In the falco alerts "Unexpected setuid call by non-sudo events contain no details" is comming all
00:40:51.862247143: Notice Unexpected setuid call by non-sudo, non-root program (container_name=host user= user_loginuid=-1 cur_uid=4294967295 parent= command= pid=1224253 uid= container_id=host image=) k8s.ns= k8s.pod= container=host
Environment