Closed jotak closed 3 months ago
Resource footprints look good: https://docs.google.com/spreadsheets/d/1qakBaK1dk_rERO30k1cSR4W-Nn0SXW4A3lqQ1sZC4rE/edit#gid=705512397 I ran two scenarios, one for dropping duplicates and another for sampling 1:50 duplicates. => both of them show around -30% CPU/memory/storage usage by Loki; FLP CPU decreased by ~12% and FLP memory decreased by 16% in drop scenario and 10% in sampling scenario.
Attention: Patch coverage is 76.92308%
with 27 lines
in your changes are missing coverage. Please review.
Project coverage is 67.41%. Comparing base (
d2b2352
) to head (fb76a92
).
Files | Patch % | Lines |
---|---|---|
pkg/api/transform_filter.go | 46.15% | 11 Missing and 3 partials :warning: |
pkg/pipeline/transform/transform_filter.go | 85.71% | 10 Missing and 3 partials :warning: |
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
Attention: Patch coverage is 76.92308%
with 27 lines
in your changes missing coverage. Please review.
Project coverage is 67.41%. Comparing base (
d2b2352
) to head (fb76a92
). Report is 27 commits behind head on main.
This has no impact on netobserv operator currently, hence marking no-qe
/approve
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: jotak
The full list of commands accepted by this bot can be found here.
The pull request process is described here
/test unit
Note this PR is based on 2 other non-merged PRs - commit https://github.com/netobserv/flowlogs-pipeline/commit/ae226f9826f46df792cd78174c8a87b80eebcafe is the significant one
Description
New "remove_entry_all_satisfied" filter type: entry is removed only if all the conditions (represented by nested rules) are satisfied. This allows to have logical AND in filter conditions, whereas previously it was only possible to have logical OR
New "conditional_sampling" filter type: allows to have random sampling based on conditions. For example, a flow matching conditions A and B may have a sampling ratio of 1:10 whereas a flow matching condition C has 1:100 sampling and all other flows are 1:1
Introduced a "preprocess" function on rules; currently it's only used to be able to cast the
Value interface{}
as an int (otherwise it comes as a float64); but could be also used in the future for other purpose, e.g. regex pre-compilingAdd tests
Breaking change
The configuration of the
removeEntryIfXXX
rules is modified, they now all share the same key for additional configuration, which is namedremoveEntry
For example, a rule previously written as:
must be changed to:
Same applies to rules of type
remove_entry_if_doesnt_exist
,remove_entry_if_equal
andremove_entry_if_not_equal
.Dependencies
n/a
Checklist
If you are not familiar with our processes or don't know what to answer in the list below, let us know in a comment: the maintainers will take care of that.