cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.07k stars 3.8k forks source link

roachtest: jepsen/multi-register/strobe-skews failed #102746

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.jepsen/multi-register/strobe-skews failed with artifacts on master @ 9e0b06bf61700d5192f171c6ebaec7e8a10fa6e6:

test artifacts and logs in: /artifacts/jepsen/multi-register/strobe-skews/run_1
(jepsen.go:434).runJepsen: timed out
(cluster.go:1410).FailOnInvalidDescriptors: invalid descriptors check failed: operation "invalid descriptors check" timed out after 1m0.013s (given timeout 1m0s): pq: query execution canceled due to statement timeout

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-27631

renatolabs commented 1 year ago

Test failed because the multi-register test timed out after 40 mins:

https://github.com/cockroachdb/cockroach/blob/2973e8345ad2b44566dd4b8a3184c9d0d965b7ef/pkg/cmd/roachtest/tests/jepsen.go#L373-L387

The stacks in invoke.log seem to indicate some threads blocked on jepsen/synchronize during test setup:

https://github.com/cockroachdb/jepsen/blob/3d7c345d6958f067edb097f9b82ab8e7a4a752c7/cockroachdb/src/jepsen/cockroach.clj#L87

This happens while the cluster is "restarted to work around balancing bug". This meant something when it was written (7 years ago), but now the context is probably either lost or known to a handful of people.

Doesn't seem to be a release blocker, so I'm closing the issue; in addition, no cockroach logs are available, so it's hard to know what happened.

cc'ing @cockroachdb/kv in case one of you has more context or want to take a look.