cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.96k stars 3.79k forks source link

roachtest: ldr/kv0/workload=both/ingestion=both failed #128276

Closed cockroach-teamcity closed 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.ldr/kv0/workload=both/ingestion=both failed with artifacts on master @ a0a52309cbbe5cb68d576fb366411fa7d7cf3574:

(sql_runner.go:99).ExecWithMessage: error executing query="SET CLUSTER SETTING kv.rangefeed.enabled = true" args=[]: dial tcp 4.246.128.190:26257: connect: connection refused
test artifacts and logs in: /artifacts/ldr/kv0/workload=both/ingestion=both/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-40908

navsetlur commented 1 month ago

As a first step, this has nothing to do with LDR. The setup was unable to reach a node and set a cluster setting

dt commented 1 month ago

n3 shutdown with code 7:

clock synchronization error: this node is more than 400ms away from at least half of the known nodes (0 of 1 are within the offset)