search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.01k stars 3.79k forks source link

roachtest: cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed #132517

Open cockroach-teamcity opened 3 days ago

cockroach-teamcity commented 3 days ago

roachtest.cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(cluster.go:2449).Run: full command output in run_061238.253388241_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-43107

cockroach-teamcity commented 1 day ago

roachtest.cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (4h0m0s)
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

wenyihu6 commented 1 day ago

Similar to other failures https://github.com/cockroachdb/cockroach/issues/132413 - splits didn't finish 

low replica RPC: have been waiting 12.85s (0 attempts) for RPC GC [/Table/106/1/‹×›,/Ta

|   | <truncated> ... 5 463 workload/workloadsql/workloadsql.go:194  [-] 53  finished 51000 of 100000 splits
  |   | I241012 06:56:47.404922 463 workload/workloadsql/workloadsql.go:194  [-] 54  finished 52000 of 100000 splits
  |   | Error: executing ALTER TABLE kv SPLIT AT VALUES (6239456554646281742): pq: split at key /Table/106/1/6239456554646281742 failed: replica unavailable: (n1,s1):5 unable to serve request to r35811:/Table/106/1/62{38903157858038340-40009951434525144} [(n1,s1):5, (n3,s3):2, (n2,s2):4, next=6, gen=72, sticky=9223372036.854775807,2147483647]: closed timestamp: 1728716124.349108045,0 (2024-10-12 06:55:24); raft status: {"id":"5","term":20,"vote":"5","commit":96,"lead":"5","raftState":"StateLeader","applied":96,"progress":{"2":{"match":0,"next":44,"state":"StateSnapshot"},"4":{"match":96,"next":119,"state":"StateReplicate"},"5":{"match":118,"next":119,"state":"StateReplicate"}},"leadtransferee":"0"}: encountered poisoned latch /Local/Range/Table/106/1/6238903157858038340/Transaction/"f93ae2e7-fb8f-4618-9916-e76ceb3ce7d0"@0,0
cockroach-teamcity commented 1 day ago

roachtest.cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(cluster.go:2449).Run: full command output in run_062408.482957187_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 23 hours ago

roachtest.cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(cluster.go:2449).Run: full command output in run_075907.930846593_n6_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 minutes ago

roachtest.cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:275
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc_bench.go:93
                                main/pkg/cmd/roachtest/test_runner.go:1279
                                src/runtime/asm_amd64.s:1700
    Error:          Received unexpected error:
                    pq: replica unavailable: (n4,s4):1 unable to serve request to r87409:/Table/106/1/397{062195564605694-246661160686828} [(n4,s4):1, (n3,s3):2, (n5,s5):3VOTER_DEMOTING_LEARNER, (n1,s1):4VOTER_INCOMING, next=5, gen=64, sticky=9223372036.854775807,2147483647]: closed timestamp: 1728979496.434909336,0 (2024-10-15 08:04:56); raft status: {"id":"1","term":6,"vote":"1","commit":159,"lead":"1","raftState":"StateLeader","applied":159,"progress":{"4":{"match":159,"next":558,"state":"StateReplicate"},"1":{"match":556,"next":558,"state":"StateReplicate"},"2":{"match":0,"next":15,"state":"StateSnapshot"},"3":{"match":556,"next":558,"state":"StateReplicate"}},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal PushTxn(PUSH_TIMESTAMP,00000000->2de05bd0) [/Local/Range/Table/106/1/397062195564605694/RangeDescriptor]
    Test:           cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!