search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

kv: All nodes in drt-chaos went down due to error "replica inconsistency was detected" #134168

Open nameisbhaskar opened 1 day ago

nameisbhaskar commented 1 day ago

drt-chaos is in a really bad state now. Several of the nodes are down, with at least two of them complaining about consistency issues: F241031 14:03:24.454366 1160675458 kv/kvserver/replica_consistency.go:832 ⋮ [T1,Vsystem,n1,s3,r2614737/1:‹/Table/643372/5/{65/7…-127/…}›] 1538467 +This node is terminating because a replica inconsistency was detected between [n1,s3,r2614737/1:‹/Table/643372/5/{65/7…-127/…}›]

Slack threads - https://cockroachlabs.slack.com/archives/C05FHJJ0MD0/p1730353029336369 https://cockroachlabs.slack.com/archives/C0KB9Q03D/p1730409574466829

Jira issue: CRDB-43952

blathers-crl[bot] commented 1 day ago

Hi @nameisbhaskar, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

blathers-crl[bot] commented 22 hours ago

Hi @ajstorm, please add branch-* labels to identify which branch(es) this GA-blocker affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

ajstorm commented 22 hours ago

Adding the GA blocker tag until we have a better idea of what's going on here.