search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

pkg/ccl/logictestccl: [result is ambiguous: error=[NotLeaseHolderError] proxy failed] TestCCLLogic_regional_by_row failed #121098

Closed cockroach-teamcity closed 7 months ago

cockroach-teamcity commented 7 months ago

pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test.TestCCLLogic_regional_by_row failed with artifacts on release-23.1.17-custom @ 215ece44cfba2ab950570f9d03cf29599e1ad088:

=== RUN   TestCCLLogic_regional_by_row
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row465789
    test_log_scope.go:81: use -show-logs to present logs inline
[06:16:06] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 19 statements
[06:16:09] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 31 statements
[06:16:15] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 38 statements
[06:16:17] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 39 statements
[06:16:23] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 43 statements
[06:16:27] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 44 statements
[06:16:32] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 51 statements
[06:16:37] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 54 statements
[06:16:47] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 56 statements
[06:16:49] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 63 statements
[06:16:51] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 66 statements
[06:16:53] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 75 statements
[06:16:56] --- progress: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row: 84 statements
    logic.go:2975: 

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row:858: USE two_region_test_db; INSERT INTO t VALUES (3) RETURNING (crdb_region)
        expected success, but found
        (40003) result is ambiguous: error=[NotLeaseHolderError] proxy failed, update client information desc: r227:/Tenant/1{0/Table/112/43/"@"/PrefixEnd-1} [(n2,s2):1, (n7,s7):2, (n5,s5):3, (n6,s6):4, (n8,s8):5, next=6, gen=158], lease: repl=(n6,s6):4 seq=5 start=1711433799.957376836,0 epo=1 pro=1711433799.971586420,0, closed_timestamp_policy: LAG_BY_CLUSTER_SETTING != et:desc:r245:/Tenant/1{0/Table/114/3/"\xc0"/PrefixEnd-1} [(n2,s2):1, (n7,s7):2, (n5,s5):3, (n6,s6):4, (n8,s8):5, next=6, gen=176], lease:repl=(n6,s6):4 seq=5 start=1711433799.957376836,0 epo=1 pro=1711433799.971586420,0; r245: replica not lease holder; current lease is repl=(n6,s6):4 seq=5 start=1711433799.957376836,0 epo=1 pro=1711433799.971586420,0 [exhausted] (last error: routing information detected to be stale: [NotLeaseHolderError] lease held by different store; r245: replica (n2,s2):1 not lease holder; current lease is repl=(n6,s6):4 seq=5 start=1711433799.957376836,0 epo=1 pro=1711433799.971586420,0)
        dist_sender.go:2352: in noMoreReplicasErr()
[06:16:57] --- done: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row with config multiregion-9node-3region-3azs-tenant: 91 tests, 1 failures
    logic.go:4145: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row:863: error while processing
    logic.go:4145: /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3298/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/logictestccl/tests/multiregion-9node-3region-3azs-tenant/multiregion-9node-3region-3azs-tenant_test_/multiregion-9node-3region-3azs-tenant_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/logictestccl/testdata/logic_test/regional_by_row:863: too many errors encountered, skipping the rest of the input
    panic.go:626: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/0451cbd40094c53fa53acd1ee2497281/logTestCCLLogic_regional_by_row465789
--- FAIL: TestCCLLogic_regional_by_row (79.13s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/sql-queries

This test on roachdash | Improve this report!

Jira issue: CRDB-37070

kvoli commented 7 months ago

Per https://github.com/cockroachdb/cockroach/issues/120944#issuecomment-2021492189

Tests shouldn't be running on this branch; closing.

Closing as unactionable.