search

bacalhau-project / bacalhau

Compute over Data framework for public, transparent, and optionally verifiable computation
https://docs.bacalhau.org
Apache License 2.0
643 stars 85 forks source link

KVMigration Tests Flaky #4052

Open wdbaruni opened 1 month ago

wdbaruni commented 1 month ago

Example: https://app.circleci.com/pipelines/github/bacalhau-project/bacalhau/12452/workflows/d6bd7527-1b89-4aad-bdaf-07c5c05a31a2/jobs/168963?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary

=== FAIL: pkg/routing/kvstore TestKVMigrationSuite/TestMigrationStoreEmpty (0.03s)
    migration_test.go:111: 
                Error Trace:    /Users/distiller/repo/pkg/routing/kvstore/migration_test.go:111
                Error:          Received unexpected error:
                                nats: API error: code=500 err_code=10049 description=error creating store for stream
                                failed to create key-value store
                                github.com/bacalhau-project/bacalhau/pkg/routing/kvstore.NewNodeStore
                                        /Users/distiller/repo/pkg/routing/kvstore/kvstore.go:55
                                github.com/bacalhau-project/bacalhau/pkg/routing/kvstore_test.(*KVMigrationSuite).TestMigrationStoreEmpty
                                        /Users/distiller/repo/pkg/routing/kvstore/migration_test.go:107
                                reflect.Value.call
                                        /usr/local/go/src/reflect/value.go:596
                                reflect.Value.Call
                                        /usr/local/go/src/reflect/value.go:380
                                github.com/stretchr/testify/suite.Run.func1
                                        /Users/distiller/go/pkg/mod/github.com/stretchr/testify@v1.9.0/suite/suite.go:202
                                testing.tRunner
                                        /usr/local/go/src/testing/testing.go:1595
                                runtime.goexit
                                        /usr/local/go/src/runtime/asm_arm64.s:1197
                Test:           TestKVMigrationSuite/TestMigrationStoreEmpty
frrist commented 1 month ago

I was unable to reproduce this on ubuntu or my macbook via go test -run KVMigrationSuite -count 500 which runs the test 500 times. Still, more flaky tests is never good.

The error code 10049 means "generic stream creation error" according to the NATs docs, which doesn't give us a lot to go on here. My gut check says this is related to CI and the temporary filesystem used by the test.

Not sure what to change here as the test is using the same setup code as other tests. How do you want to test things that use NATS @wdbaruni?