search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachtest: jepsen/sequential/split failed #130410

Closed cockroach-teamcity closed 2 months ago

cockroach-teamcity commented 2 months ago

roachtest.jepsen/sequential/split failed with artifacts on master @ 7571549d8772a087ab4577d1e770e02582ba5877:

(test_runner.go:1325).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/jepsen/sequential/split/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-42044

renatolabs commented 2 months ago

The Jepsen test passed but, for some reason, we grab certain artifacts even when the test passes. It then timed out because scp was stuck for over 40 minutes:

2024/09/10 12:03:55 cluster_synced.go:2424: teamcity-16816736-1725947306-77-n6cpu4: getting (scp) /mnt/data1/jepsen/cockroachdb/store/latest/results.edn /artifacts/jepsen/sequential/split/cpu_arch=arm64/run_1/results.edn on nodes [6]
2024/09/10 12:43:37 test_impl.go:423: test failure #1: full stack retained in failure_1.log: (test_runner.go:1325).runTest: test timed out (3h0m0s)

This is somewhat similar to #130279, where scp was also stuck for a really long time.

Closing.