search

cockroachdb / cockroach

CockroachDB - the open source, cloud-native distributed SQL database.
https://www.cockroachlabs.com
Other
29.5k stars 3.7k forks source link

kv/kvserver: TestTenantRateLimiter failed #125728

Open cockroach-teamcity opened 2 weeks ago

cockroach-teamcity commented 2 weeks ago

kv/kvserver.TestTenantRateLimiter failed on release-24.1.1-rc @ 484f8e2f4d3cb23582416944d79eab72ff42a6a1:

                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestTenantRateLimiter
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_tenant_test.go:257
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) aborted in DistSender
                            Wraps: (3) result is ambiguous: context deadline exceeded
                            Wraps: (4) secondary error attachment
                              | aborted during DistSender.Send: context deadline exceeded
                              | (1)
                              |   | (opaque error wrapper)
                              |   | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
                              |   | reportable 0:
                              |   |
                              |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:3128
                              |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2091
                              |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).divideAndSendBatchToRanges
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1616
                              |   | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).Send
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1232
                              |   | github.com/cockroachdb/cockroach/pkg/kv.(*CrossRangeTxnWrapperSender).Send
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:229
                              |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).sendUsingSender
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1138
                              |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).send
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:1121
                              |   | github.com/cockroachdb/cockroach/pkg/kv.sendAndFill
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:970
                              |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Run
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:993
                              |   | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Put
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/db.go:426
                              |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver_test.TestTenantRateLimiter
                              |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver_test/pkg/kv/kvserver/client_tenant_test.go:257
                              |   | testing.tRunner
                              |   |     GOROOT/src/testing/testing.go:1689
                              |   | runtime.goexit
                              |   |     src/runtime/asm_amd64.s:1695
                              | Wraps: (2) aborted during DistSender.Send
                              | Wraps: (3) context deadline exceeded
                              | Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) context.deadlineExceededError
                            Wraps: (5) context deadline exceeded
                            Error types: (1) *errbase.opaqueWrapper (2) *errutil.withPrefix (3) *kvpb.AmbiguousResultError (4) *secondary.withSecondaryError (5) context.deadlineExceededError
            Test:           TestTenantRateLimiter
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestTenantRateLimiter1010213813
--- FAIL: TestTenantRateLimiter (120.78s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-39574

arulajmani commented 2 weeks ago

@rickystewart this test seems to be running into overload issues consistently when run on EngFlow. Is there something we can tweak here to prevent this?