netobserv / netobserv-ebpf-agent

Network Observability eBPF Agent
Apache License 2.0
127 stars 32 forks source link

avoid racing with kernelspace in LookupAndDelete #282

Closed jotak closed 7 months ago

openshift-ci[bot] commented 7 months ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please ask for approval from jotak. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/netobserv/netobserv-ebpf-agent/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
codecov[bot] commented 7 months ago

Codecov Report

Attention: Patch coverage is 0% with 4 lines in your changes are missing coverage. Please review.

Project coverage is 36.05%. Comparing base (d4d25a4) to head (2816933).

Files Patch % Lines
pkg/ebpf/tracer.go 0.00% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #282 +/- ## ========================================== - Coverage 36.08% 36.05% -0.03% ========================================== Files 42 42 Lines 3786 3789 +3 ========================================== Hits 1366 1366 - Misses 2342 2345 +3 Partials 78 78 ``` | [Flag](https://app.codecov.io/gh/netobserv/netobserv-ebpf-agent/pull/282/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/netobserv/netobserv-ebpf-agent/pull/282/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv) | `36.05% <0.00%> (-0.03%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=netobserv#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

jotak commented 7 months ago

My theory for this change is that during a LookupAndDelete iteration, kernel space continues to receive packets for the flows already deleted, add them to the map, forcing user space to read the same flow again, hence increasing the iteration, hence giving more time for kernel to again add flows and make the process repeated indefinitely. So, better skip the flow ids that are already processed.

However I cannot say that the theory was confirmed from tests. The processing time for LookupAndDelete is still very high with this patch. Mem & cpu profiles are changing quite a lot but hard to say if it's positive or negative: cf https://docs.google.com/spreadsheets/d/1qakBaK1dk_rERO30k1cSR4W-Nn0SXW4A3lqQ1sZC4rE/edit#gid=1192055209 : I see much improved memory footprint, but also much less flows processed (I don't know why) and CPU much better in low traffic but quite higher in high traffic...

image

jotak commented 7 months ago

superseded by https://github.com/netobserv/netobserv-ebpf-agent/pull/283