search

netobserv / netobserv-ebpf-agent

Network Observability eBPF Agent
Apache License 2.0
138 stars 34 forks source link

"The agent could not be able to start eBPF programs" on OCP 4.10 on top of IBM Rocks #37

Open andresmareca-ibm opened 2 years ago

andresmareca-ibm commented 2 years ago

We (me and @eranra) have managed to start all the necessary pods to be able to monitor the network following the following procedure.

  1. Clone newtork-observability-operator
  2. run make deploy ocp-deploy
  3. run oc apply -f config/samples/flows_v1alpha1_flowcollector.yaml
  4. Set ebpf-agent pods as privileged as is described here: https://github.com/netobserv/netobserv-ebpf-agent/issues/34

Now onnetobserv-ebpf-agent-* pods I get the following error message:

time="2022-06-23T11:04:51Z" level=info msg="starting NetObserv eBPF Agent"
time="2022-06-23T11:04:51Z" level=info msg="initializing Flows agent" component=agent.Flows
time="2022-06-23T11:04:51Z" level=info msg="push CTRL+C or send SIGTERM to interrupt execution"
time="2022-06-23T11:04:51Z" level=info msg="starting Flows agent" component=agent.Flows
time="2022-06-23T11:04:51Z" level=warning msg="can't remove mem lock. The agent could not be able to start eBPF programs" component=systemSetup error="unexpected error detecting memory cgroup accounting: permission denied"
time="2022-06-23T11:04:51Z" level=info msg="Flows agent successfully started" component=agent.Flows
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=eth0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=eth0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=eth1
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=eth1
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=vethlocal
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=vethlocal
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=tunl0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=tunl0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib83c121d54f
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib83c121d54f
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali4ffc7d0c2cd
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali4ffc7d0c2cd
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia361f7a2ef0
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia361f7a2ef0
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali2d403d4ded4
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali2d403d4ded4
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali08aea38f7a5
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali08aea38f7a5
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali5132e419805
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali5132e419805
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib5e3a5f56f6
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib5e3a5f56f6
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif6df68b6a97
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif6df68b6a97
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali24dc76da138
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali24dc76da138
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali7ede1555ef2
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali7ede1555ef2
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calib26e009b516
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calib26e009b516
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali9eea980de3d
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali9eea980de3d
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali4458a90e5f5
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali4458a90e5f5
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali209bf37fc41
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali209bf37fc41
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali988f8eb4584
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali988f8eb4584
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali62d9ccb9b76
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali62d9ccb9b76
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali724c631a936
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali724c631a936
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=califfa03e79339
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=califfa03e79339
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali9c18e337f00
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali9c18e337f00
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calic38c45648fd
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calic38c45648fd
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali435d7b6f56c
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali435d7b6f56c
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif2d31cc4040
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif2d31cc4040
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calif0d73f1499c
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calif0d73f1499c
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia32c3fcf987
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia32c3fcf987
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali0a8626f2f08
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali0a8626f2f08
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali090d631fef7
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali090d631fef7
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali49f1199c7c6
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali49f1199c7c6
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calice4e6974a40
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calice4e6974a40
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali7c05cd50d45
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali7c05cd50d45
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=cali57e83446543
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=cali57e83446543
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calibc77efe6d8a
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calibc77efe6d8a
time="2022-06-23T11:04:51Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calicf3cf0bb3bb
time="2022-06-23T11:04:51Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calicf3cf0bb3bb
time="2022-06-23T11:04:55Z" level=info msg="interface detected. Registering flow tracer" component=agent.Flows name=calia8a8ea72ccb
time="2022-06-23T11:04:55Z" level=warning msg="can't register flow tracer. Ignoring" component=agent.Flows error="loading and assigning BPF objects: field EgressFlowParse: program egress_flow_parse: map .rodata: map create: read- and write-only maps not supported (requires >= v5.2)" interface=calia8a8ea72ccb

If I check the kernel version it shows me the following:

sh-4.4# chroot /host
sh-4.2# uname -a
Linux kube-caop3qnf0asm47p248i0-testocp410-default-00000121.iks.ibm 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed Apr 27 20:34:34 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Has it happened to any of you?

eranra commented 2 years ago

@mariomac @praveingk please your help ^^^

mariomac commented 2 years ago

@andresmareca-ibm This kernel version is 3.10, which seems way too old to run eBPF according to: https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

We are supporting version 4.18+, which is the kernel version supported by RHEL 8, although some earlier versions could also work, but 3.10 seems too old even for basic eBPF support, according to the documentation.

eranra commented 2 years ago

@elevran FYI kernel version is 3.10 so whatever we talked about with regards to alternatives to eBPF using he kernel is not relevant for such an old kernel :-(