Open cockroach-teamcity opened 2 weeks ago
cli.TestDebugCheckStore failed on release-24.1 @ 9400571541f006110b28dac7780e91dcec85140d:
E240502 06:25:26.645703 3299576 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r32/1:/{Table/29-NamespaceTab…}] 1787 [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29] E240502 06:25:26.646083 3299577 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r17/1:/Table/1{4-5}] 1788 [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14] E240502 06:25:26.646522 3299588 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r52/1:/Table/5{0-1}] 1789 [n1,s1,r52/1:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/50] E240502 06:25:26.646900 3299589 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r24/1:/Table/2{1-2}] 1790 [n1,s1,r24/1:/Table/2{1-2}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r24:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"0","commit":46,"lead":"0","raftState":"StatePreCandidate","applied":46,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/21] E240502 06:25:26.647255 3299590 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r56/1:/Table/5{4-5}] 1791 [n1,s1,r56/1:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54] E240502 06:25:26.647867 3299601 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r56/2:/Table/5{4-5}] 1792 [n3,s3,r56/2:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54] E240502 06:25:26.648723 3299612 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r52/1:/Table/5{0-1}] 1793 [n1,s1,r52/1:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/50] E240502 06:25:26.649113 3299613 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r24/1:/Table/2{1-2}] 1794 [n1,s1,r24/1:/Table/2{1-2}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r24:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"0","commit":46,"lead":"0","raftState":"StatePreCandidate","applied":46,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/21] E240502 06:25:26.649506 3299614 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r56/1:/Table/5{4-5}] 1795 [n1,s1,r56/1:/Table/5{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r56:/Table/5{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":108,"lead":"0","raftState":"StatePreCandidate","applied":108,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/54] E240502 06:25:26.650216 3299635 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r28/1:/Table/2{5-6}] 1796 [n1,s1,r28/1:/Table/2{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r28:/Table/2{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630869.076533290,0 (2024-05-02 06:21:09); raft status: {"id":"1","term":11,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/25] E240502 06:25:26.650617 3299636 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r48/1:/Table/4{5-6}] 1797 [n1,s1,r48/1:/Table/4{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r48:/Table/4{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630855.425441632,0 (2024-05-02 06:20:55); raft status: {"id":"1","term":11,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/45] E240502 06:25:26.651489 3299647 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r28/1:/Table/2{5-6}] 1798 [n1,s1,r28/1:/Table/2{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r28:/Table/2{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630869.076533290,0 (2024-05-02 06:21:09); raft status: {"id":"1","term":11,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/25] E240502 06:25:26.651876 3299648 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r48/1:/Table/4{5-6}] 1799 [n1,s1,r48/1:/Table/4{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r48:/Table/4{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630855.425441632,0 (2024-05-02 06:20:55); raft status: {"id":"1","term":11,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/45] E240502 06:25:26.652656 3299659 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r14/2:/Table/1{1-2}] 1800 [n3,s3,r14/2:/Table/1{1-2}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r14:/Table/1{1-2} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"1","commit":176,"lead":"0","raftState":"StatePreCandidate","applied":176,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/11] E240502 06:25:26.653230 3299660 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r47/1:/Table/4{4-5}] 1801 [n1,s1,r47/1:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/44] E240502 06:25:26.653885 3299681 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r32/2:/{Table/29-NamespaceTab…}] 1802 [n3,s3,r32/2:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"2","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/29] E240502 06:25:26.654523 3299682 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r47/1:/Table/4{4-5}] 1803 [n1,s1,r47/1:/Table/4{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r47:/Table/4{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/44] E240502 06:25:26.655089 3299693 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r20/2:/Table/1{7-8}] 1804 [n3,s3,r20/2:/Table/1{7-8}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r20:/Table/1{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"2","term":8,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/17] E240502 06:25:26.655651 3299694 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r9/1:/Table/{5-6}] 1805 [n1,s1,r9/1:/Table/{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r9:/Table/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630872.740566404,0 (2024-05-02 06:21:12); raft status: {"id":"1","term":10,"vote":"1","commit":58,"lead":"0","raftState":"StatePreCandidate","applied":58,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/5] E240502 06:25:26.656213 3299705 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r35/2:/Table/3{2-3}] 1806 [n3,s3,r35/2:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":8,"vote":"2","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/32] E240502 06:25:26.658274 3299706 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r35/1:/Table/3{2-3}] 1807 [n1,s1,r35/1:/Table/3{2-3}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r35:/Table/3{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":8,"vote":"2","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/32] E240502 06:25:26.658708 3299748 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r21/1:/Table/1{8-9}] 1808 [n1,s1,r21/1:/Table/1{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r21:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.304740579,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":6,"vote":"1","commit":40,"lead":"0","raftState":"StatePreCandidate","applied":40,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/18] E240502 06:25:26.659083 3299749 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r9/1:/Table/{5-6}] 1809 [n1,s1,r9/1:/Table/{5-6}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r9:/Table/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630872.740566404,0 (2024-05-02 06:21:12); raft status: {"id":"1","term":10,"vote":"1","commit":58,"lead":"0","raftState":"StatePreCandidate","applied":58,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/5] E240502 06:25:26.659939 3299760 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r31/2:/Table/2{8-9}] 1810 [n3,s3,r31/2:/Table/2{8-9}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r31:/Table/2{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"2","term":8,"vote":"2","commit":45,"lead":"0","raftState":"StatePreCandidate","applied":45,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/28] E240502 06:25:26.660557 3299727 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r7/1:/Table/{3-4}] 1811 [n1,s1,r7/1:/Table/{3-4}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"1","term":7,"vote":"1","commit":47,"lead":"0","raftState":"StatePreCandidate","applied":47,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/3] E240502 06:25:26.660958 3299771 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r17/1:/Table/1{4-5}] 1812 [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14] E240502 06:25:26.661402 3299772 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 1813 [n1,s1,r3/1:/System/{NodeLive…-tsd}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.427326243,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":114,"lead":"0","raftState":"StatePreCandidate","applied":114,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/System/NodeLivenessMax] E240502 06:25:26.662263 3299783 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r52/2:/Table/5{0-1}] 1814 [n3,s3,r52/2:/Table/5{0-1}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r52:/Table/5{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630868.464415262,0 (2024-05-02 06:21:08); raft status: {"id":"2","term":8,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/50] E240502 06:25:26.662830 3299784 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r7/1:/Table/{3-4}] 1815 [n1,s1,r7/1:/Table/{3-4}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r7:/Table/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630866.804846702,0 (2024-05-02 06:21:06); raft status: {"id":"1","term":7,"vote":"1","commit":47,"lead":"0","raftState":"StatePreCandidate","applied":47,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/3] E240502 06:25:26.663521 3299805 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r4/2:/System{/tsd-tse}] 1816 [n3,s3,r4/2:/System{/tsd-tse}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r4:/System{/tsd-tse} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630860.708105018,0 (2024-05-02 06:21:00); raft status: {"id":"2","term":9,"vote":"1","commit":99,"lead":"0","raftState":"StatePreCandidate","applied":99,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/System/tsd] E240502 06:25:26.664195 3299806 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r8/2:/Table/{4-5}] 1817 [n3,s3,r8/2:/Table/{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r8:/Table/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"2","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/4] E240502 06:25:26.664805 3299817 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r12/1:/Table/{8-9}] 1818 [n1,s1,r12/1:/Table/{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r12:/Table/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/8] E240502 06:25:26.665180 3299818 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r51/1:/Table/{48-50}] 1819 [n1,s1,r51/1:/Table/{48-50}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r51:/Table/{48-50} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630856.908159549,0 (2024-05-02 06:20:56); raft status: {"id":"1","term":11,"vote":"1","commit":37,"lead":"0","raftState":"StatePreCandidate","applied":37,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/48] E240502 06:25:26.665573 3299819 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r61/1:/Table/{59-60}] 1820 [n1,s1,r61/1:/Table/{59-60}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r61:/Table/{59-60} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/59] E240502 06:25:26.666147 3299830 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r17/2:/Table/1{4-5}] 1821 [n3,s3,r17/2:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"2","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/14] --- FAIL: TestDebugCheckStore (329.93s) E240502 06:25:26.672598 3299831 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r25/1:/Table/2{2-3}] 1822 [n1,s1,r25/1:/Table/2{2-3}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r25:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n3,s3):2,(n2,s2):3); closed timestamp: 1714630871.518659348,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":8,"vote":"1","commit":49,"lead":"0","raftState":"StatePreCandidate","applied":49,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/22] E240502 06:25:26.672998 3299832 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r17/1:/Table/1{4-5}] 1823 [n1,s1,r17/1:/Table/1{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r17:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630857.424347936,0 (2024-05-02 06:20:57); raft status: {"id":"1","term":9,"vote":"1","commit":36,"lead":"0","raftState":"StatePreCandidate","applied":36,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/14] E240502 06:25:26.673447 3299833 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r3/1:/System/{NodeLive…-tsd}] 1824 [n1,s1,r3/1:/System/{NodeLive…-tsd}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.427326243,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":114,"lead":"0","raftState":"StatePreCandidate","applied":114,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/System/NodeLivenessMax] E240502 06:25:26.673838 3299834 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r12/1:/Table/{8-9}] 1825 [n1,s1,r12/1:/Table/{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r12:/Table/{8-9} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":7,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal RequestLease [/Table/8] E240502 06:25:26.674224 3299835 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r51/1:/Table/{48-50}] 1826 [n1,s1,r51/1:/Table/{48-50}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r51:/Table/{48-50} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630856.908159549,0 (2024-05-02 06:20:56); raft status: {"id":"1","term":11,"vote":"1","commit":37,"lead":"0","raftState":"StatePreCandidate","applied":37,"progress":{},"leadtransferee":"0"}: have been waiting 61.00s for slow proposal RequestLease [/Table/48] E240502 06:25:26.674598 3299836 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r61/1:/Table/{59-60}] 1827 [n1,s1,r61/1:/Table/{59-60}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r61:/Table/{59-60} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630868.243609667,0 (2024-05-02 06:21:08); raft status: {"id":"1","term":7,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/59] E240502 06:25:26.674980 3299837 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r32/1:/{Table/29-NamespaceTab…}] 1828 [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29] E240502 06:25:26.675337 3299838 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r58/1:/Table/5{6-7}] 1829 [n1,s1,r58/1:/Table/5{6-7}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r58:/Table/5{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"1","term":8,"vote":"1","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/56] E240502 06:25:26.675879 3299839 kv/kvserver/queue.go:1202 [T1,Vsystem,n3,replicate,s3,r68/2:/{Table/66-Max}] 1830 [n3,s3,r68/2:/{Table/66-Max}]: could not obtain lease: replica unavailable: (n3,s3):2 unable to serve request to r68:/{Table/66-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n1,s1):1); closed timestamp: 1714630867.829180892,0 (2024-05-02 06:21:07); raft status: {"id":"2","term":8,"vote":"2","commit":39,"lead":"0","raftState":"StatePreCandidate","applied":39,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/66] E240502 06:25:26.676678 3299840 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r41/1:/Table/3{8-9}] 1831 [n1,s1,r41/1:/Table/3{8-9}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r41:/Table/3{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]: lost quorum (down: (n2,s2):2,(n3,s3):3); closed timestamp: 1714630871.478771401,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":7,"vote":"2","commit":53,"lead":"0","raftState":"StatePreCandidate","applied":53,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Table/38] E240502 06:25:26.677381 3299851 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r37/1:/Table/3{4-5}] 1832 [n1,s1,r37/1:/Table/3{4-5}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r37:/Table/3{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630875.315373110,0 (2024-05-02 06:21:15); raft status: {"id":"1","term":7,"vote":"2","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal RequestLease [/Table/34] E240502 06:25:26.677815 3299852 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,replicate,s1,r32/1:/{Table/29-NamespaceTab…}] 1833 [n1,s1,r32/1:/{Table/29-NamespaceTab…}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r32:/{Table/29-NamespaceTable/30} [(n1,s1):1, (n3,s3):2, next=3, gen=2]: lost quorum (down: (n3,s3):2); closed timestamp: 1714630873.745492719,0 (2024-05-02 06:21:13); raft status: {"id":"1","term":10,"vote":"1","commit":42,"lead":"0","raftState":"StatePreCandidate","applied":42,"progress":{},"leadtransferee":"0"}: have been waiting 63.00s for slow proposal RequestLease [/Table/29] E240502 06:25:26.678561 3299863 kv/kvserver/queue.go:1202 [T1,Vsystem,n1,lease,s1,r23/1:/Table/2{0-1}] 1834 [n1,s1,r23/1:/Table/2{0-1}]: could not obtain lease: replica unavailable: (n1,s1):1 unable to serve request to r23:/Table/2{0-1} [(n1,s1):1, (n2,s2):2, next=3, gen=2]: lost quorum (down: (n2,s2):2); closed timestamp: 1714630871.304740579,0 (2024-05-02 06:21:11); raft status: {"id":"1","term":6,"vote":"1","commit":41,"lead":"0","raftState":"StatePreCandidate","applied":41,"progress":{},"leadtransferee":"0"}: have been waiting 62.50s for slow proposal RequestLease [/Table/20] E240502 06:25:26.679071 3299864 kv/kvserver/queue.go:1202 [T1,Vsystem,n2,replicaGC,s2,r67/2:/Table/6{5-6}] 1835 replica unavailable: (n2,s2):3 unable to serve request to r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n3,s3):2, (n2,s2):3, next=4, gen=4]: lost quorum (down: (n1,s1):1,(n3,s3):2); closed timestamp: 1714630871.298175213,0 (2024-05-02 06:21:11); raft status: {"id":"3","term":7,"vote":"0","commit":336,"lead":"0","raftState":"StatePreCandidate","applied":336,"progress":{},"leadtransferee":"0"}: have been waiting 60.50s for slow proposal RequestLease [/Min]
Parameters:
attempt=1
run=16
shard=6
See also: How To Investigate a Go Test Failure (internal)
- #122328 cli: TestDebugCheckStore failed [C-bug C-test-failure O-robot P-3 T-kv branch-master]
/cc @cockroachdb/kv @cockroachdb/server
This test on roachdash | Improve this report!
Jira issue: CRDB-38357
dupe of #112219 - assigning P3 label.
cli.TestDebugCheckStore failed on release-24.1 @ 9400571541f006110b28dac7780e91dcec85140d:
Parameters:
attempt=1
run=16
shard=6
Help
See also: How To Investigate a Go Test Failure (internal)
Same failure on other branches
- #122328 cli: TestDebugCheckStore failed [C-bug C-test-failure O-robot P-3 T-kv branch-master]
/cc @cockroachdb/kv @cockroachdb/server
This test on roachdash | Improve this report!
Jira issue: CRDB-38357