TestL7NetworkPolicy e2e tests failing consistently

[antoninbas]

Describe the bug I have seen the TestL7NetworkPolicy e2e tests fail consistently for a few recent PRs in Github CI. All the subtests are marked as failed, not a specific one. Example of a failed test: https://github.com/antrea-io/antrea/actions/runs/8365150919/job/22902786915 Expand the section below for the full output.

Full test output

``` 2024-03-20T20:51:02.7716987Z === RUN TestL7NetworkPolicy 2024-03-20T20:51:02.7717806Z 2024/03/20 20:51:02 Applying Antrea YAML 2024-03-20T20:51:04.2231461Z 2024/03/20 20:51:04 Waiting for all Antrea DaemonSet Pods 2024-03-20T20:51:05.2300461Z 2024/03/20 20:51:05 Checking CoreDNS deployment 2024-03-20T20:51:05.2319945Z fixtures.go:260: Creating 'testl7networkpolicy-po20rlkw' K8s Namespace 2024-03-20T20:51:05.3742943Z I0320 20:51:05.373896 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:51:05.4269252Z I0320 20:51:05.426512 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-kqcnr' 2024-03-20T20:51:05.6261831Z I0320 20:51:05.625891 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:51:05.6834810Z I0320 20:51:05.683139 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-n45jp' 2024-03-20T20:51:05.8582958Z I0320 20:51:05.857893 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:51:05.9204027Z I0320 20:51:05.920064 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-v6zsb' 2024-03-20T20:51:21.0456648Z === RUN TestL7NetworkPolicy/HTTP 2024-03-20T20:51:25.0664974Z === RUN TestL7NetworkPolicy/HTTP/Ingress 2024-03-20T20:51:25.0666763Z l7networkpolicy_test.go:121: Creating ANNP test-l7-http-allow-path-hostname 2024-03-20T20:51:25.0766651Z l7networkpolicy_test.go:121: Creating ANNP test-l7-http-allow-any-path 2024-03-20T20:51:32.0842075Z l7networkpolicy_test.go:131: 2024-03-20T20:51:32.0843511Z Error Trace: /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:131 2024-03-20T20:51:32.0845673Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:247 2024-03-20T20:51:32.0846441Z Error: Condition never satisfied 2024-03-20T20:51:32.0847100Z Test: TestL7NetworkPolicy/HTTP/Ingress 2024-03-20T20:51:41.4391008Z === RUN TestL7NetworkPolicy/HTTP/Egress 2024-03-20T20:51:41.4393345Z l7networkpolicy_test.go:121: Creating ANNP test-l7-http-allow-path-hostname 2024-03-20T20:51:41.4465603Z l7networkpolicy_test.go:121: Creating ANNP test-l7-http-allow-any-path 2024-03-20T20:51:48.4533157Z l7networkpolicy_test.go:131: 2024-03-20T20:51:48.4534350Z Error Trace: /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:131 2024-03-20T20:51:48.4536091Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:274 2024-03-20T20:51:48.4537253Z Error: Condition never satisfied 2024-03-20T20:51:48.4538182Z Test: TestL7NetworkPolicy/HTTP/Egress 2024-03-20T20:51:55.7755564Z === RUN TestL7NetworkPolicy/TLS 2024-03-20T20:51:58.7859616Z l7networkpolicy_test.go:121: Creating ANNP test-l7-tls-allow-sni-google 2024-03-20T20:51:58.7910802Z l7networkpolicy_test.go:121: Creating ANNP test-l7-tls-allow-sni-facebook 2024-03-20T20:52:05.7987747Z l7networkpolicy_test.go:173: 2024-03-20T20:52:05.7989740Z Error Trace: /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:173 2024-03-20T20:52:05.7991991Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:323 2024-03-20T20:52:05.7994137Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:60 2024-03-20T20:52:05.7995427Z Error: Condition never satisfied 2024-03-20T20:52:05.7996262Z Test: TestL7NetworkPolicy/TLS 2024-03-20T20:52:10.7994194Z l7networkpolicy_test.go:173: 2024-03-20T20:52:10.7995620Z Error Trace: /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:173 2024-03-20T20:52:10.7997925Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:324 2024-03-20T20:52:10.8000493Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:60 2024-03-20T20:52:10.8001813Z Error: Condition never satisfied 2024-03-20T20:52:10.8002902Z Test: TestL7NetworkPolicy/TLS 2024-03-20T20:52:13.3406879Z l7networkpolicy_test.go:179: Expected not to access the server, but the request succeeded. 2024-03-20T20:52:13.3407765Z Stdout: 2024-03-20T20:52:13.3408544Z 2024-03-20T20:52:13.3419612Z 2024-03-20T20:52:13.3430271Z 2024-03-20T20:52:13.3432135Z 2024-03-20T20:52:13.3433984Z 2024-03-20T20:52:13.3435831Z 2024-03-20T20:52:13.3531162Z

Update Your Browser

You’re using a web browser that isn’t supported by Facebook.
To get a better experience, go to one of these sites and get the latest version of your preferred browser:

Google Chrome

Mozilla Firefox

Get Facebook on Your Phone

Stay connected anytime, anywhere.

2024-03-20T20:52:13.3693177Z 2024-03-20T20:52:13.3756830Z 2024-03-20T20:52:13.3769756Z 2024-03-20T20:52:13.3779533Z 2024-03-20T20:52:13.3781448Z 2024-03-20T20:52:13.3783165Z 2024-03-20T20:52:13.3784954Z 2024-03-20T20:52:13.3786603Z 2024-03-20T20:52:13.3788029Z 2024-03-20T20:52:13.3789719Z 2024-03-20T20:52:13.3791462Z 2024-03-20T20:52:13.3793486Z 2024-03-20T20:52:13.3795481Z 2024-03-20T20:52:13.3798403Z 2024-03-20T20:52:13.3801570Z 2024-03-20T20:52:13.3829007Z 2024-03-20T20:52:13.3846308Z Stderr: Connecting to www.facebook.com (31.13.66.35:443) 2024-03-20T20:52:13.3846979Z Connecting to www.facebook.com (31.13.66.35:443) 2024-03-20T20:52:13.3847517Z writing to stdout 2024-03-20T20:52:13.3848049Z - 9485 --:--:-- ETA 2024-03-20T20:52:13.3848759Z - 100% |********************************| 44190 0:00:00 ETA 2024-03-20T20:52:13.3849361Z written to stdout 2024-03-20T20:52:13.3849731Z 2024-03-20T20:52:17.8045339Z l7networkpolicy_test.go:173: 2024-03-20T20:52:17.8046938Z Error Trace: /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:173 2024-03-20T20:52:17.8049011Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:330 2024-03-20T20:52:17.8050641Z /home/runner/work/antrea/antrea/test/e2e/l7networkpolicy_test.go:60 2024-03-20T20:52:17.8052530Z Error: Condition never satisfied 2024-03-20T20:52:17.8053522Z Test: TestL7NetworkPolicy/TLS 2024-03-20T20:52:20.8388901Z I0320 20:52:20.838251 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:52:20.8994900Z I0320 20:52:20.898892 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-9jbqt' 2024-03-20T20:52:21.1385513Z I0320 20:52:21.138085 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:52:21.1952922Z I0320 20:52:21.194769 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-cf2fm' 2024-03-20T20:52:21.3773904Z I0320 20:52:21.376921 24505 framework.go:2656] Sending SIGINT to 'antrea-agent-coverage' 2024-03-20T20:52:21.4401101Z I0320 20:52:21.439717 24505 framework.go:2662] Copying coverage files from Pod 'antrea-agent-f5llg' 2024-03-20T20:52:35.5773120Z === NAME TestL7NetworkPolicy 2024-03-20T20:52:35.5774548Z fixtures.go:333: Exporting test logs to '/home/runner/work/antrea/antrea/log/TestL7NetworkPolicy/beforeTeardown.Mar20-20-52-35' 2024-03-20T20:52:38.2668726Z fixtures.go:504: Deleting 'testl7networkpolicy-po20rlkw' K8s Namespace 2024-03-20T20:52:38.2707734Z I0320 20:52:38.270542 24505 framework.go:793] Deleting Namespace testl7networkpolicy-po20rlkw took 3.977318ms 2024-03-20T20:52:38.2709519Z --- FAIL: TestL7NetworkPolicy (95.50s) 2024-03-20T20:52:38.2710717Z --- FAIL: TestL7NetworkPolicy/HTTP (34.73s) 2024-03-20T20:52:38.2711873Z --- FAIL: TestL7NetworkPolicy/HTTP/Ingress (14.37s) 2024-03-20T20:52:38.2713063Z --- FAIL: TestL7NetworkPolicy/HTTP/Egress (14.34s) 2024-03-20T20:52:38.2719080Z --- FAIL: TestL7NetworkPolicy/TLS (24.93s) ```

[antoninbas]

@tnqn the only recent related change I can think of is the logging fix, but that should not have impacted NP enforcement

[tnqn]

If it started two days ago, https://github.com/antrea-io/antrea/pull/4537 may be related.

The current log doesn't provide any infromation as all agents were restarted before collecting the logs. We need to invert the order to know what happened.

@hongliangl could you check this one?

[hongliangl]

If it started two days ago, #4537 may be related.

The current log doesn't provide any infromation as all agents were restarted before collecting the logs. We need to invert the order to know what happened.

@hongliangl could you check this one?

Will do