Falco 0.36.2 BPF probe conflict with deepflow

Spartan-65 commented 11 months ago

Describe the bug Run deepflow and falco ebpf mode

If you start deepflow first, and then start falco, falco will report the following logs

Defaulted container "falco" out of: falco, falcoctl-artifact-follow, falco-driver-loader (init), falcoctl-artifact-install (init)
Fri Dec  8 03:11:16 2023: Falco version: 0.36.2 (x86_64)
Fri Dec  8 03:11:16 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Dec  8 03:11:16 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Dec  8 03:11:16 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Fri Dec  8 03:11:16 2023: Starting health webserver with threadiness 8, listening on port 8765
Fri Dec  8 03:11:16 2023: Loaded event sources: syscall
Fri Dec  8 03:11:16 2023: Enabled event sources: syscall
Fri Dec  8 03:11:16 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o
Syscall event drop monitoring:
- event drop detected: 0 occurrences
- num times actions taken: 0
03:11:22.806934827: Notice Unexpected connection to K8s API Server from container (connection=100.22.0.1:43340->100.22.0.1:443 lport=443 rport=43340 fd_type=ipv4 fd_proto=fd.l4proto evt_type=connect user=root user_uid=0 user_loginuid=-1 process=daemon proc_exepath=/daemon parent=supervisord command=daemon terminal=0 exe_flags=<NA> container_id=e2cf2fdbadbe container_image=192.168.2.40:8080/tmp/daemon container_image_tag=2.20.0 container_name=k8s_daemon_tensorsec-holmes-kjpqq_tensorsec_a5472cb2-a6d9-468d-a020-c85f538ebe09_0 k8s_ns=tensorsec k8s_pod_name=tensorsec-holmes-kjpqq)
Events detected: 1
Rule counts by severity:
NOTICE: 1
Triggered rules by rule name:
Contact K8S API Server From Container: 1
Error: scap_next buffer corruption

If you start falco first, then deepflow, the host will go down.
If falco is running in kernel module mode, both will work correctly

How to reproduce it

Expected behaviour

Screenshots

Environment

Falco version:

root@falco-f4wz6:/# falco --version
Fri Dec  8 04:17:43 2023: Falco version: 0.36.2 (x86_64)
Fri Dec  8 04:17:43 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Falco version: 0.36.2
Libs version:  0.13.4
Plugin API:    3.1.0
Engine:        26
Driver:
API version:    5.0.0
Schema version: 2.0.0
Default driver: 6.0.1+driver

System info:


root@falco-4tl44:/# falco --support | jq .system_info
Fri Dec  8 04:19:10 2023: Falco version: 0.36.2 (x86_64)
Fri Dec  8 04:19:10 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Dec  8 04:19:10 2023: Loading rules from file /etc/falco/falco_rules.yaml
{
"machine": "x86_64",
"nodename": "falco-4tl44",
"release": "4.19.25-204.el7.bclinux.x86_64",
"sysname": "Linux",
"version": "#1 SMP Wed Dec 23 15:41:17 CST 2020"
}

<!-- Falco has a built-in support command you can use  "falco --support | jq .system_info" -->
- Cloud provider or hardware configuration:
- OS:

NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos"


<!-- Eg., output of "cat /etc/os-release". -->
- Kernel:

Linux falco-cgq72 4.19.25-204.el7.bclinux.x86_64 #1 SMP Wed Dec 23 15:41:17 CST 2020 x86_64 GNU/Linux

<!-- Eg., output of "uname -a". -->
- Installation method:
<!-- Eg., Kubernetes, RPM, DEB, from source? -->

Kubernetes


**Additional context**

 I found that the kernel module also uses this function "static inline int32_t ringbuffer_next(struct scap_device_set *devset, OUT scap_evt** pevent, OUT uint16_t* pcpuid)
", why does it only get an error in ebpf mode?

<!-- Add any other context about the problem here. -->

Andreagit97 commented 11 months ago

uhm this is interesting it seems like the 2 ebpf probe are conflicting in some way,

Error: scap_next buffer corruption

this message means that someone wrote in the falco bpf buffer :thinking: we will take a look, thank you for reporting!

Andreagit97 commented 11 months ago

I've seen you are using 0.36.2 with the kernel module, I'm not able to reproduce this issue https://github.com/falcosecurity/libs/issues/1359 with Falco 0.36.2 can you confirm this?

poiana commented 8 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

Spartan-65 commented 7 months ago

I tried to insert the log at this location

if(pe->ts < min_ts)
        {
            printf("pe->len: %u, dev->m_sn_len: %u dev->m_lastreadsize: %u\n", pe->len, dev->m_sn_len, dev->m_lastreadsize);
            /* if the event length is greater than the remaining size in our block there is something wrong! */
            if(pe->len > dev->m_sn_len)
            {
                snprintf(devset->m_lasterr, SCAP_LASTERR_SIZE, "scap_next buffer corruption");

                /* if you get the following assertion, first recompile the driver and `libscap` */
                ASSERT(false);
                return SCAP_FAILURE;
            }

Got some of the output

pe->len: 3207276032, dev->m_sn_len: 1176 dev->m_lastreadsize: 0
pe->len: 3207276032, dev->m_sn_len: 1176 dev->m_lastreadsize: 0
pe->len: 3207276032, dev->m_sn_len: 1176 dev->m_lastreadsize: 0

the pe->len value seems to be significantly larger than SCRATCH_SIZE_MAX (1<<18 - 1) source code.

Can it be assumed that deepflow wrote to the bpf perf map or that Falco misread the corresponding BPF map?

Spartan-65 commented 7 months ago

deepflow ebpf map info

 bpftool map
1: hash  name calico_failsafe  flags 0x1
    key 4B  value 1B  max_entries 65535  memlock 5246976B
36: hash  name __active_read_a  flags 0x0
    key 8B  value 60B  max_entries 40960  memlock 5967872B
37: hash  name __active_write_  flags 0x0
    key 8B  value 60B  max_entries 40960  memlock 5967872B
38: array  name __adapt_kern_ui  flags 0x0
    key 4B  value 8B  max_entries 1  memlock 4096B
39: percpu_array  name __ctx_info  flags 0x0
    key 4B  value 176B  max_entries 1  memlock 4096B
40: percpu_array  name __data_buf  flags 0x0
    key 4B  value 32768B  max_entries 1  memlock 266240B
41: percpu_array  name __http2_stack  flags 0x0
    key 4B  value 8334B  max_entries 1  memlock 69632B
42: percpu_array  name __io_event_buff  flags 0x0
    key 4B  value 80B  max_entries 1  memlock 4096B
43: array  name __kprobe_port_b  flags 0x0
    key 4B  value 8192B  max_entries 2  memlock 20480B
44: percpu_array  name __members_offse  flags 0x0
    key 4B  value 80B  max_entries 1  memlock 4096B
45: prog_array  name __progs_jmp_kp_  flags 0x0
    key 4B  value 4B  max_entries 2  memlock 4096B
46: prog_array  name __progs_jmp_tp_  flags 0x0
    key 4B  value 4B  max_entries 3  memlock 4096B
47: array  name __protocol_filt  flags 0x0
    key 4B  value 4B  max_entries 130  memlock 4096B
48: perf_event_array  name __socket_data  flags 0x0
    key 4B  value 4B  max_entries 256  memlock 4096B
49: hash  name __socket_info_m  flags 0x0
    key 8B  value 58B  max_entries 524288  memlock 71307264B
50: hash  name __socket_role_m  flags 0x0
    key 8B  value 4B  max_entries 40960  memlock 3674112B
51: hash  name __ssl_ctx_map  flags 0x0
    key 8B  value 20B  max_entries 40960  memlock 4329472B
52: percpu_array  name __trace_conf_ma  flags 0x0
    key 4B  value 48B  max_entries 1  memlock 4096B
53: hash  name __trace_map  flags 0x0
    key 16B  value 25B  max_entries 524288  memlock 58724352B
54: array  name __trace_stats_m  flags 0x0
    key 4B  value 16B  max_entries 1  memlock 4096B
55: lru_hash  name go_ancerstor_ma  flags 0x0
    key 12B  value 8B  max_entries 40960  memlock 4001792B
56: lru_hash  name go_rw_ts_map  flags 0x0
    key 12B  value 8B  max_entries 40960  memlock 4001792B
57: hash  name goroutines_map  flags 0x0
    key 8B  value 8B  max_entries 40960  memlock 3674112B
58: lru_hash  name http2_tcp_seq_m  flags 0x0
    key 12B  value 4B  max_entries 40960  memlock 4001792B
59: hash  name pid_tgid_caller  flags 0x0
    key 8B  value 16B  max_entries 40960  memlock 4001792B
60: hash  name proc_info_map  flags 0x0
    key 4B  value 64B  max_entries 40960  memlock 5967872B
61: hash  name tls_conn_map  flags 0x0
    key 16B  value 32B  max_entries 40960  memlock 4984832B
62: perf_event_array  name __profiler_outp  flags 0x0
    key 4B  value 4B  max_entries 256  memlock 4096B
63: perf_event_array  name __profiler_outp  flags 0x0
    key 4B  value 4B  max_entries 256  memlock 4096B
64: array  name __profiler_stat  flags 0x0
    key 4B  value 8B  max_entries 7  memlock 4096B
65: stack_trace  name __stack_map_a  flags 0x0
    key 4B  value 1016B  max_entries 65536  memlock 68161536B
66: stack_trace  name __stack_map_b  flags 0x0
    key 4B  value 1016B  max_entries 65536  memlock 68161536B

Spartan-65 commented 6 months ago

I've seen you are using 0.36.2 with the kernel module, I'm not able to reproduce this issue falcosecurity/libs#1359 with Falco 0.36.2 can you confirm this?

@Andreagit97 It's not the same environment, it's in a user environment where deepflowis deployed

Andreagit97 commented 6 months ago

uhm ok! thank you for the additional info! Yes at first look it seems that deepflow wrote into Falco perf buffer... we will take a look! Thank you for reporting. Is this happening also with the modern_ebpf?

poiana commented 5 months ago

Stale issues rot after 30d of inactivity.

Mark the issue as fresh with /remove-lifecycle rotten.

Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle rotten

Andreagit97 commented 5 months ago

/remove-lifecycle rotten

poiana commented 2 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

FedeDP commented 2 months ago

/remove-lifecycle stale

falcosecurity / falco

Falco 0.36.2 BPF probe conflict with deepflow #2957