CI flakes in regression gateway test on RouteOption

artberger commented 3 years ago

Describe the bug

From the logs:

...
{"level":"info","ts":1629401253.8677585,"logger":"proxy-validator","caller":"validation/server.go:190","msg":"proxy validation report result: listener_reports:{http_listener_report:{virtual_host_reports:{route_reports:{}}}} listener_reports:{http_listener_report:{}}"} {"level":"info","ts":1629401254.845505,"logger":"proxy-validator","caller":"validation/server.go:179","msg":"received proxy validation request"} {"level":"info","ts":1629401254.8462656,"logger":"proxy-validator.translator","caller":"translator/translator.go:177","msg":"computing envoy resources for listener: listener-::-8080"} {"level":"info","ts":1629401254.8465867,"logger":"proxy-validator.translator","caller":"translator/translator.go:177","msg":"computing envoy resources for listener: listener-::-8443"} {"level":"info","ts":1629401254.8477402,"logger":"proxy-validator","caller":"validation/server.go:190","msg":"proxy validation report result: listener_reports:{http_listener_report:{virtual_host_reports:{route_reports:{}}}} listener_reports:{http_listener_report:{}}"}
LOGS FROM gloo-system.testrunner: STARTING SLEEP! Access me.
** End Kubernetes Dump ** 
[AfterEach] tests with RouteOptions
  /home/runner/work/gloo/gloo/test/kube2e/gateway/gateway_test.go:960
[AfterEach] Kube2e: gateway
  /home/runner/work/gloo/gloo/test/kube2e/gateway/gateway_test.go:186
make: *** [run-tests] Error 1
SSSSSSSS
JUnit report was created: /home/runner/work/gloo/gloo/test/kube2e/gateway/junit.xml

Summarizing 1 Failure:

[Fail] Kube2e: gateway tests with RouteOptions [It] correctly delegates options from RouteOption 
/home/runner/go/pkg/mod/github.com/solo-io/go-utils@v0.21.16/testutils/assertions.go:27

Ran 13 of 21 Specs in 183.525 seconds
FAIL! -- 12 Passed | 1 Failed | 0 Pending | 8 Skipped
--- FAIL: TestGateway (183.81s)
FAIL

Ginkgo ran 2 suites in 14m46.640050239s
Test Suite Failed
Makefile:148: recipe for target 'run-tests' failed
Error: Process completed with exit code 2.

To Reproduce Steps to reproduce the behavior:

Open a PR in the repo and it might fail (?)

Expected behavior Pass the regression test.

Additional context

Hit this initially in this PR. Sam mentioned it's popped up a few times, so opening this issue here in case others run into it.

MitchAman commented 3 years ago

https://github.com/solo-io/gloo/pull/5190/checks?check_run_id=3402122862

sam-heilbron commented 3 years ago

Again on https://github.com/solo-io/gloo/pull/5261

artberger commented 3 years ago

https://github.com/solo-io/gloo/pull/5446/checks?check_run_id=3819642653#step:11:762

sam-heilbron commented 3 years ago

This is a frequent test flake that plagues developers. We identified this as one of the noisier tests. Timebox to 1-2 days. If no progress, re-sync with the team.

gunnar-solo commented 2 years ago

failure found on a backport to 1.9: https://github.com/solo-io/gloo/runs/4597385507?check_suite_focus=true

artberger commented 2 years ago

Another instance: https://github.com/solo-io/gloo/runs/5289121368?check_suite_focus=true

Summarizing 1 Failure:

[Fail] Kube2e: gateway tests with RouteOptions [It] correctly delegates options from RouteOption 
/home/runner/go/pkg/mod/github.com/solo-io/go-utils@v0.21.24/testutils/assertions.go:27

Ran 17 of 31 Specs in 219.963 seconds
FAIL! -- 16 Passed | 1 Failed | 0 Pending | 14 Skipped
--- FAIL: TestGateway (220.51s)
FAIL

Ginkgo ran 1 suite in 19m58.128322212s
Test Suite Failed
make: *** [run-tests] Error 1
Makefile:148: recipe for target 'run-tests' failed
Error: Process completed with exit code 2.

sam-heilbron commented 2 years ago

https://github.com/solo-io/gloo/runs/5604489908?check_suite_focus=true

sam-heilbron commented 2 years ago

https://github.com/solo-io/gloo/runs/5742236269?check_suite_focus=true

sam-heilbron commented 2 years ago

This has been merged into 1.12 -> 1.9.

solo-io / gloo

CI flakes in regression gateway test on RouteOption #5202