search

onsi / ginkgo

A Modern Testing Framework for Go
http://onsi.github.io/ginkgo/
MIT License
8.38k stars 660 forks source link

internal/internal_integration/node_spec_events_test.go:33 may be flaky #1452

Open siretart opened 3 months ago

siretart commented 3 months ago

While looking at the debian package of ginkgo, I can across https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1078516 that indicates that the Integration Tests may be flaky. Here are two instances where the same test fails in a very similar manner:

Usually this passes on a retry.

The test in question appears to be here: https://github.com/onsi/ginkgo/blob/master/internal/internal_integration/node_spec_events_test.go#L33-L70

It is possible that this test is affected by heavy load on the host, I couldn't reproduce it yet on my laptop.

onsi commented 3 months ago

Thanks for reporting this - I don't know who/how the debian packages are maintained but it seems odd to me that installing ginkgo ends up running all the tests. There's no need to run these and they (especially the integration tests run in series) will add several minutes to the build time. Do you know where we could go to change how ginkgo is built and installed?

siretart commented 3 months ago

Happy to clarify as needed.

Installing the ginkgo package does not run any tests. The tests are being run as part of the build process (and there is another way to execute them using the autopkgtest machinery, but that's unrelated to this bug).

As you can see from the build logs above, this is how the tests are being invoked:

    cd build && go test -vet=off -v -p 8 -timeout=1h github.com/onsi/ginkgo/v2 github.com/onsi/ginkgo/v2/config github.com/onsi/ginkgo/v2/docs github.com/onsi/ginkgo/v2/dsl github.com/onsi/ginkgo/v2/dsl/core github.com/onsi/ginkgo/v2/dsl/decorators github.com/onsi/ginkgo/v2/dsl/reporting github.com/onsi/ginkgo/v2/dsl/table github.com/onsi/ginkgo/v2/extensions/globals github.com/onsi/ginkgo/v2/extensions/table github.com/onsi/ginkgo/v2/formatter github.com/onsi/ginkgo/v2/ginkgo github.com/onsi/ginkgo/v2/ginkgo/build github.com/onsi/ginkgo/v2/ginkgo/command github.com/onsi/ginkgo/v2/ginkgo/generators github.com/onsi/ginkgo/v2/ginkgo/internal github.com/onsi/ginkgo/v2/ginkgo/labels github.com/onsi/ginkgo/v2/ginkgo/outline github.com/onsi/ginkgo/v2/ginkgo/performance github.com/onsi/ginkgo/v2/ginkgo/run github.com/onsi/ginkgo/v2/ginkgo/unfocus github.com/onsi/ginkgo/v2/ginkgo/watch github.com/onsi/ginkgo/v2/integration github.com/onsi/ginkgo/v2/internal github.com/onsi/ginkgo/v2/internal/global github.com/onsi/ginkgo/v2/internal/internal_integration github.com/onsi/ginkgo/v2/internal/interrupt_handler github.com/onsi/ginkgo/v2/internal/parallel_support github.com/onsi/ginkgo/v2/internal/test_helpers github.com/onsi/ginkgo/v2/internal/testingtproxy github.com/onsi/ginkgo/v2/reporters github.com/onsi/ginkgo/v2/types

the long list of packages is dynamically generated with go list. If you feel that some of those packages are inappropriate to run as part of the package build, I can easily exclude them. Please just let me know.

onsi commented 3 months ago

ah, thanks for clarifying. yes this is likely due to a slow build machine. The test is a bit too finnicky about the measured duration so I'm pushing a fix that will make it more lenient (I generally try to avoid testing code with actual time.Sleeps but in this case with a black-box integration test it's the cleanest approach). I'll but a new release with the fix soon.

as for the invocation. you can use go test but I would recommend using the ginkgo cli to run the tests. more recent versions of ginkgo ship with a makefile so you could just run make test.

onsi commented 3 months ago

v2.20.1 has the fix. I don't generally backport bugfixes to prior minor releases, however, so this will not resolve flakiness issues on older version of Ginkgo.

siretart commented 3 months ago

Thank you so much!

I've backported your commit as https://salsa.debian.org/go-team/packages/golang-github-onsi-ginkgo-v2/-/commit/107e820 so that it can be included in the next package update.

onsi commented 3 months ago

sounds good. let me know if you see it flake again (i wasn't able to reproduce it locally but am pretty sure this is the issue).

siretart commented 2 months ago

I finally got around to actually upload this change to debian, only to find another timeout on armel: https://ci.debian.net/packages/g/golang-github-onsi-ginkgo-v2/testing/armel/51760224/

726s Emitting Node SpecEvents [It] emits each spec event as it goes
726s /tmp/autopkgtest-lxc.kf8bgbeu/downtmp/autopkgtest_tmp/build/src/github.com/onsi/ginkgo/v2/internal/internal_integration/node_spec_events_test.go:72
726s 
726s   [FAILED] Timeline failed to match:
726s 
726s     Failed at index 5:
726s           <types.SpecEvent>: [Node (End) SpecEvent] Message="A",Duration=35.765684ms,NodeType=It,CL=/tmp/autopkgtest-lxc.kf8bgbeu/downtmp/autopkgtest_tmp/build/src/github.com/onsi/ginkgo/v2/internal/internal_integration/node_spec_events_test.go:19,TL.Offset=6
726s   Value for field 'Duration' failed to satisfy matcher.
726s   Expected
726s       <time.Duration>: 35765684
726s   to be within 10ms of ~
726s       <time.Duration>: 20000000
726s   
726s   In [It] at: /tmp/autopkgtest-lxc.kf8bgbeu/downtmp/autopkgtest_tmp/build/src/github.com/onsi/ginkgo/v2/internal/internal_integration/node_spec_events_test.go:77 @ 09/16/24 15:44:08.618
726s ------------------------------
726s •••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••••
726s 
726s Summarizing 1 Failure:
726s   [FAIL] Emitting Node SpecEvents [It] emits each spec event as it goes

At least the issue originally in this bugreport didn't get triggered.

I can file a separate issue, if you prefer. let me know.

Thanks!

onsi commented 3 weeks ago

hey sorry for the long latency. i've bumped this threshold as well and will cut another release oon