cilium / tetragon

eBPF-based Security Observability and Runtime Enforcement
https://tetragon.io
Apache License 2.0
3.67k stars 370 forks source link

ci: e2e tests unmatched events does not work as expected #3063

Open kkourt opened 3 weeks ago

kkourt commented 3 weeks ago

In some recent CI failures, it seemed that even though only one event was unmatched, the unmatched JSON output included all events. The only way to diagnose this was to add -test.v to the e2e test args. For reference, I'm attaching the logs afer enabling -test.v and artifacts from the same run.

In the logs, you can see that only one event was missed:

2024-10-31T08:59:08.8668768Z time="2024-10-31T08:59:08Z" level=info msg="UnorderedEventChecker: checking pending 1/1: ProcessExecChecker/otel-demo-opensearch: Process check failed: ProcessChecker: Pod check failed: PodChecker: PodLabels[app.kubernetes.io/component] (app.kubernetes.io/component=loadgenerator) check failed: 'loadgenerator' does not match full 'otel-demo-opensearch'"
2024-10-31T08:59:08.8672504Z time="2024-10-31T08:59:08Z" level=info msg="PROCESS_EXEC:3584 => no match: UnorderedEventChecker: all 1 check(s) failed, continuing"
2024-10-31T08:59:09.0824913Z I1031 08:59:09.081023   13892 rpcchecker.go:249] event checker labelsEventChecker timed out after 10m0s

But the in the unmatched json event, all events are unmatched:

jq -r '.[] | to_entries[] | .value.checkerName' < labelsEventChecker.eventchecker.unmatched.json
otel-demo-grafana
otel-demo-jaeger
otel-demo-otelcol
otel-demo-prometheus
otel-demo-accountingservice
otel-demo-adservice
otel-demo-cartservice
otel-demo-checkoutservice
otel-demo-currencyservice
otel-demo-emailservice
otel-demo-flagd
otel-demo-frauddetectionservice
otel-demo-frontend
otel-demo-frontendproxy
otel-demo-kafka
otel-demo-loadgenerator
otel-demo-paymentservice
otel-demo-productcatalogservice
otel-demo-quoteservice
otel-demo-recommendationservice
otel-demo-redis
otel-demo-shippingservice
otel-demo-opensearch

labelsEventChecker.eventchecker.events.json labelsEventChecker.eventchecker.log labelsEventChecker.eventchecker.unmatched.json 4ubuntu-22.04-arm64 labels.txt