cockroach-teamcity
commented
4 months ago upgrade/upgrades.TestLeasingClusterVersionStarvation failed on master @ 2f498d0abcf4130e5c872b73200086d6a3af9f02:
Fatal error:
panic: test timed out after 14m57s
running tests:
TestLeasingClusterVersionStarvation (14m43s)
Stack:
goroutine 2113744 [running]:
testing.(*M).startAlarm.func1()
GOROOT/src/testing/testing.go:2366 +0x385
created by time.goFunc
GOROOT/src/time/sleep.go:177 +0x2dLog preceding fatal error
``` I240508 04:50:45.044512 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r19/1:/Table/1{6-7}] 1582 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.044673 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r13/1:/Table/{9-11}] 1583 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.044840 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r30/1:/Table/2{7-8}] 1584 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045006 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r64/1:/{Table/62-Max}] 1585 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045184 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r14/1:/Table/1{1-2}] 1586 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045386 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 1587 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045551 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r27/1:/Table/2{4-5}] 1588 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045716 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r29/1:/Table/2{6-7}] 1589 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.045886 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r15/1:/Table/1{2-3}] 1590 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046049 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r37/1:/Table/3{4-5}] 1591 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046208 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r1/1:/{Min-System/NodeL…}] 1592 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046381 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r42/1:/Table/{39-40}] 1593 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046545 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r35/1:/Table/3{2-3}] 1594 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046707 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r23/1:/Table/2{0-1}] 1595 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.046890 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r11/1:/Table/{7-8}] 1596 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.047053 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r26/1:/Table/2{3-4}] 1597 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.047213 9077 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r24/1:/Table/2{1-2}] 1598 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240508 04:50:45.047308 8831 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1599 gossip status (ok, 1 node) I240508 04:50:45.047308 8831 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1599 +gossip client (0/3 cur/max conns) I240508 04:50:45.047308 8831 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1599 +gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received) I240508 04:50:45.200760 8834 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1600 ={"Timestamp":1715143845200757553,"EventType":"runtime_stats","MemRSSBytes":1465716736,"GoroutineCount":316,"MemStackSysBytes":4587520,"GoAllocBytes":1024336760,"GoTotalBytes":1191875112,"HeapFragmentBytes":17357960,"HeapReservedBytes":120995840,"HeapReleasedBytes":69926912,"CGoAllocBytes":135652560,"CGoTotalBytes":141352960,"CGoCallRate":48457.94,"CPUUserPercent":75.90226,"CPUSysPercent":24.000713,"GCPausePercent":0.000028160837,"GCRunCount":206,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240508 04:50:45.200943 8834 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1601 runtime stats: 1.4 GiB RSS, 316 goroutines (stacks: 4.4 MiB), 977 MiB/1.1 GiB Go alloc/total (heap fragmentation: 17 MiB, heap reserved: 115 MiB, heap released: 67 MiB), 129 MiB/135 MiB CGO alloc/total (48457.9 CGO/sec), 75.9/24.0 %(u/s)time, 0.0 %gc (206x), 416 B/416 B (r/w)net I240508 04:50:55.201007 8834 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1602 ={"Timestamp":1715143855201003930,"EventType":"runtime_stats","MemRSSBytes":1465737216,"GoroutineCount":318,"MemStackSysBytes":4325376,"GoAllocBytes":1032806424,"GoTotalBytes":1191875112,"HeapFragmentBytes":18784232,"HeapReservedBytes":111362048,"HeapReleasedBytes":69926912,"CGoAllocBytes":135685328,"CGoTotalBytes":141393920,"CGoCallRate":48833.29,"CPUUserPercent":77.19809,"CPUSysPercent":22.699438,"GCPausePercent":0.00039679016,"GCRunCount":207,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240508 04:50:55.201178 8834 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1603 runtime stats: 1.4 GiB RSS, 318 goroutines (stacks: 4.1 MiB), 985 MiB/1.1 GiB Go alloc/total (heap fragmentation: 18 MiB, heap reserved: 106 MiB, heap released: 67 MiB), 129 MiB/135 MiB CGO alloc/total (48833.3 CGO/sec), 77.2/22.7 %(u/s)time, 0.0 %gc (207x), 416 B/416 B (r/w)net I240508 04:51:02.216582 8804 3@pebble/event.go:828 [n1,s1,pebble] 1604 [JOB 60] WAL created 000070 (recycled 000032) I240508 04:51:02.224566 2037962 3@pebble/event.go:780 [n1,s1,pebble] 1605 [JOB 61] flushing 1 memtable (64MB) to L0 I240508 04:51:02.224708 2037962 3@pebble/event.go:808 [n1,s1,pebble] 1606 [JOB 61] flushing: sstable created 000071 I240508 04:51:02.540249 2037962 3@pebble/event.go:808 [n1,s1,pebble] 1607 [JOB 61] flushing: sstable created 000072 I240508 04:51:02.554571 2037962 3@pebble/event.go:784 [n1,s1,pebble] 1608 [JOB 61] flushed 1 memtable (64MB) to L0 [000071 000072] (2.8MB), in 0.3s (0.3s total), output rate 8.4MB/s I240508 04:51:02.554731 2038434 3@pebble/event.go:768 [n1,s1,pebble] 1609 [JOB 62] compacting(default) L0 [000071] (2.2MB) Score=6.62 + L6 [000066] (4.1MB) Score=0.15; OverlappingRatio: Single 1.91, Multi 0.00 I240508 04:51:02.555451 2038434 3@pebble/event.go:808 [n1,s1,pebble] 1610 [JOB 62] compacting: sstable created 000073 W240508 04:51:02.739697 10243 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:44799,class=system,rpc] 1611 latency jump (prev avg 85.37ms, current 219.30ms) I240508 04:51:03.272284 2038434 3@pebble/event.go:808 [n1,s1,pebble] 1612 [JOB 62] compacting: sstable created 000074 I240508 04:51:03.282504 2038434 3@pebble/event.go:772 [n1,s1,pebble] 1613 [JOB 62] compacted(default) L0 [000071] (2.2MB) Score=6.62 + L6 [000066] (4.1MB) Score=0.15 -> L6 [000073 000074] (4.5MB), in 0.7s (0.7s total), output rate 6.2MB/s I240508 04:51:03.282685 2039174 3@pebble/event.go:768 [n1,s1,pebble] 1614 [JOB 64] compacting(default) L0 [000072] (626KB) Score=6.38 + L6 [000067 000068 000069] (5.5MB) Score=0.16; OverlappingRatio: Single 9.08, Multi 0.00 I240508 04:51:03.283048 2039174 3@pebble/event.go:808 [n1,s1,pebble] 1615 [JOB 64] compacting: sstable created 000075 I240508 04:51:03.304251 8703 3@pebble/event.go:812 [n1,s1,pebble] 1616 [JOB 62] sstable deleted 000066 I240508 04:51:03.304360 8703 3@pebble/event.go:812 [n1,s1,pebble] 1617 [JOB 62] sstable deleted 000071 I240508 04:51:03.613970 2039174 3@pebble/event.go:808 [n1,s1,pebble] 1618 [JOB 64] compacting: sstable created 000076 I240508 04:51:03.763670 2039174 3@pebble/event.go:772 [n1,s1,pebble] 1619 [JOB 64] compacted(default) L0 [000072] (626KB) Score=6.38 + L6 [000067 000068 000069] (5.5MB) Score=0.16 -> L6 [000075 000076] (6.0MB), in 0.5s (0.5s total), output rate 13MB/s I240508 04:51:03.769442 8703 3@pebble/event.go:812 [n1,s1,pebble] 1620 [JOB 64] sstable deleted 000067 I240508 04:51:03.769527 8703 3@pebble/event.go:812 [n1,s1,pebble] 1621 [JOB 64] sstable deleted 000068 I240508 04:51:03.769574 8703 3@pebble/event.go:812 [n1,s1,pebble] 1622 [JOB 64] sstable deleted 000069 I240508 04:51:03.769632 8703 3@pebble/event.go:812 [n1,s1,pebble] 1623 [JOB 64] sstable deleted 000072 I240508 04:51:05.201030 8834 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1624 ={"Timestamp":1715143865201027417,"EventType":"runtime_stats","MemRSSBytes":1475989504,"GoroutineCount":304,"MemStackSysBytes":3932160,"GoAllocBytes":1040193408,"GoTotalBytes":1201754664,"HeapFragmentBytes":17705088,"HeapReservedBytes":115326976,"HeapReleasedBytes":60047360,"CGoAllocBytes":135674064,"CGoTotalBytes":141737984,"CGoCallRate":53588.773,"CPUUserPercent":77.39982,"CPUSysPercent":22.499947,"GCPausePercent":0.0006425585,"GCRunCount":208} I240508 04:51:05.201206 8834 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1625 runtime stats: 1.4 GiB RSS, 304 goroutines (stacks: 3.8 MiB), 992 MiB/1.1 GiB Go alloc/total (heap fragmentation: 17 MiB, heap reserved: 110 MiB, heap released: 57 MiB), 129 MiB/135 MiB CGO alloc/total (53588.8 CGO/sec), 77.4/22.5 %(u/s)time, 0.0 %gc (208x), 0 B/0 B (r/w)net I240508 04:51:15.205075 8834 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1626 ={"Timestamp":1715143875205072636,"EventType":"runtime_stats","MemRSSBytes":1475551232,"GoroutineCount":310,"MemStackSysBytes":3670016,"GoAllocBytes":1049523136,"GoTotalBytes":1202278952,"HeapFragmentBytes":18893888,"HeapReservedBytes":105070592,"HeapReleasedBytes":60047360,"CGoAllocBytes":135657680,"CGoTotalBytes":140820480,"CGoCallRate":73837.03,"CPUUserPercent":81.46704,"CPUSysPercent":18.49252,"GCPausePercent":0.0007932791,"GCRunCount":210,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240508 04:51:15.205303 8834 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1627 runtime stats: 1.4 GiB RSS, 310 goroutines (stacks: 3.5 MiB), 1001 MiB/1.1 GiB Go alloc/total (heap fragmentation: 18 MiB, heap reserved: 100 MiB, heap released: 57 MiB), 129 MiB/134 MiB CGO alloc/total (73837.0 CGO/sec), 81.5/18.5 %(u/s)time, 0.0 %gc (210x), 416 B/416 B (r/w)net I240508 04:51:25.205423 8834 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1628 ={"Timestamp":1715143885205420531,"EventType":"runtime_stats","MemRSSBytes":1470541824,"GoroutineCount":317,"MemStackSysBytes":4259840,"GoAllocBytes":1037505896,"GoTotalBytes":1197290024,"HeapFragmentBytes":20974232,"HeapReservedBytes":109428736,"HeapReleasedBytes":65036288,"CGoAllocBytes":135657680,"CGoTotalBytes":140808192,"CGoCallRate":74471.11,"CPUUserPercent":81.49716,"CPUSysPercent":18.399359,"GCPausePercent":0.0003967862,"GCRunCount":211,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240508 04:51:25.205590 8834 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1629 runtime stats: 1.4 GiB RSS, 317 goroutines (stacks: 4.1 MiB), 989 MiB/1.1 GiB Go alloc/total (heap fragmentation: 20 MiB, heap reserved: 104 MiB, heap released: 62 MiB), 129 MiB/134 MiB CGO alloc/total (74471.1 CGO/sec), 81.5/18.4 %(u/s)time, 0.0 %gc (211x), 416 B/416 B (r/w)net ```
Parameters:
attempt=1run=1shard=4Help
celiala
fqazi
rafiss
upgrade/upgrades.TestLeasingClusterVersionStarvation failed on master @ 6e23d32dd0d36181f5c886c9a0eda9c756005bd6:
Fatal error:
Stack:
Log preceding fatal error
``` I240504 07:06:53.710677 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r49/1:/Table/4{6-7}] 1637 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.710849 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r38/1:/Table/3{5-6}] 1638 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711041 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r47/1:/Table/4{4-5}] 1639 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711220 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r36/1:/Table/3{3-4}] 1640 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711400 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r51/1:/Table/{48-57}] 1641 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711567 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r12/1:/Table/{8-9}] 1642 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711741 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r60/1:/Table/{58-60}] 1643 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.711921 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r62/1:/Table/6{0-2}] 1644 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712100 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r42/1:/Table/{39-40}] 1645 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712272 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r37/1:/Table/3{4-5}] 1646 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712448 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r46/1:/Table/4{3-4}] 1647 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712615 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r22/1:/Table/{19-20}] 1648 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712793 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r43/1:/Table/4{0-1}] 1649 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.712976 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r16/1:/Table/1{3-4}] 1650 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.713147 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r48/1:/Table/4{5-6}] 1651 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.713314 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r35/1:/Table/3{2-3}] 1652 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.713484 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r9/1:/Table/{5-6}] 1653 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.713660 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r14/1:/Table/1{1-2}] 1654 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.713834 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r50/1:/Table/4{7-8}] 1655 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714039 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r4/1:/System{/tsd-tse}] 1656 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714214 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] 1657 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714395 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r30/1:/Table/2{7-8}] 1658 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714578 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r27/1:/Table/2{4-5}] 1659 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714765 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r41/1:/Table/3{8-9}] 1660 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.714961 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r10/1:/Table/{6-7}] 1661 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.715137 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r26/1:/Table/2{3-4}] 1662 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.715336 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] 1663 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.715507 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r20/1:/Table/1{7-8}] 1664 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.715692 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r1/1:/{Min-System/NodeL…}] 1665 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.761697 8923 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1666 gossip status (ok, 1 node) I240504 07:06:53.761697 8923 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1666 +gossip client (0/3 cur/max conns) I240504 07:06:53.761697 8923 2@gossip/gossip.go:531 [T1,Vsystem,n1] 1666 +gossip server (0/3 cur/max conns, infos 0/0 sent/received, bytes 0B/0B sent/received) I240504 07:06:53.777302 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r8/1:/Table/{4-5}] 1667 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:53.777579 9162 13@kv/kvserver/replicate_queue.go:767 [T1,Vsystem,n1,replicate,s1,r5/1:/{Systemtse-Table/0}] 1668 error processing replica: 0 of 1 live stores are able to take a new replica for the range (1 already has a voter, 0 already have a non-voter); likely not enough nodes in cluster I240504 07:06:54.028933 8926 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1669 ={"Timestamp":1714806414028930178,"EventType":"runtime_stats","MemRSSBytes":1108086784,"GoroutineCount":304,"MemStackSysBytes":3375104,"GoAllocBytes":697078248,"GoTotalBytes":825780696,"HeapFragmentBytes":53308952,"HeapReservedBytes":45260800,"HeapReleasedBytes":27140096,"CGoAllocBytes":134907200,"CGoTotalBytes":140455936,"CGoCallRate":197.22318,"CPUUserPercent":99.56122,"CPUSysPercent":0.2998832,"GCPausePercent":0.0005706577,"GCRunCount":194,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240504 07:06:54.029120 8926 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1670 runtime stats: 1.0 GiB RSS, 304 goroutines (stacks: 3.2 MiB), 665 MiB/788 MiB Go alloc/total (heap fragmentation: 51 MiB, heap reserved: 43 MiB, heap released: 26 MiB), 129 MiB/134 MiB CGO alloc/total (197.2 CGO/sec), 99.6/0.3 %(u/s)time, 0.0 %gc (194x), 416 B/416 B (r/w)net W240504 07:06:54.805042 10344 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=system,rpc] 1671 latency jump (prev avg 79.07ms, current 139.47ms) I240504 07:07:04.052398 8926 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1672 ={"Timestamp":1714806424052394293,"EventType":"runtime_stats","MemRSSBytes":1121226752,"GoroutineCount":303,"MemStackSysBytes":3637248,"GoAllocBytes":656402496,"GoTotalBytes":825780696,"HeapFragmentBytes":48469952,"HeapReservedBytes":90513408,"HeapReleasedBytes":27140096,"CGoAllocBytes":134921536,"CGoTotalBytes":140500992,"CGoCallRate":351.67484,"CPUUserPercent":99.566376,"CPUSysPercent":0.39906362,"GCPausePercent":0.00039587112,"GCRunCount":195,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240504 07:07:04.052579 8926 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1673 runtime stats: 1.0 GiB RSS, 303 goroutines (stacks: 3.5 MiB), 626 MiB/788 MiB Go alloc/total (heap fragmentation: 46 MiB, heap reserved: 86 MiB, heap released: 26 MiB), 129 MiB/134 MiB CGO alloc/total (351.7 CGO/sec), 99.6/0.4 %(u/s)time, 0.0 %gc (195x), 416 B/416 B (r/w)net W240504 07:07:05.641901 10331 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=default,rpc] 1674 latency jump (prev avg 79.68ms, current 132.42ms) W240504 07:07:12.195400 10344 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=system,rpc] 1675 latency jump (prev avg 77.10ms, current 172.04ms) I240504 07:07:14.073245 8926 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1676 ={"Timestamp":1714806434073241347,"EventType":"runtime_stats","MemRSSBytes":1165410304,"GoroutineCount":305,"MemStackSysBytes":3506176,"GoAllocBytes":656738024,"GoTotalBytes":868772312,"HeapFragmentBytes":52295960,"HeapReservedBytes":129474560,"HeapReleasedBytes":30285824,"CGoAllocBytes":134954304,"CGoTotalBytes":140611584,"CGoCallRate":384.8976,"CPUUserPercent":99.592384,"CPUSysPercent":0.3991679,"GCPausePercent":0.0004777241,"GCRunCount":196} I240504 07:07:14.073431 8926 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1677 runtime stats: 1.1 GiB RSS, 305 goroutines (stacks: 3.3 MiB), 626 MiB/828 MiB Go alloc/total (heap fragmentation: 50 MiB, heap reserved: 124 MiB, heap released: 29 MiB), 129 MiB/134 MiB CGO alloc/total (384.9 CGO/sec), 99.6/0.4 %(u/s)time, 0.0 %gc (196x), 0 B/0 B (r/w)net W240504 07:07:23.929911 10344 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=system,rpc] 1678 latency jump (prev avg 56.88ms, current 137.00ms) I240504 07:07:24.073303 8926 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1679 ={"Timestamp":1714806444073297386,"EventType":"runtime_stats","MemRSSBytes":1165963264,"GoroutineCount":303,"MemStackSysBytes":3309568,"GoAllocBytes":718878728,"GoTotalBytes":868772312,"HeapFragmentBytes":54626296,"HeapReservedBytes":65200128,"HeapReleasedBytes":30285824,"CGoAllocBytes":134954304,"CGoTotalBytes":140947456,"CGoCallRate":209.69884,"CPUUserPercent":99.59945,"CPUSysPercent":0.19999889,"GCPausePercent":0.0003967978,"GCRunCount":197,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240504 07:07:24.073512 8926 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1680 runtime stats: 1.1 GiB RSS, 303 goroutines (stacks: 3.2 MiB), 686 MiB/828 MiB Go alloc/total (heap fragmentation: 52 MiB, heap reserved: 62 MiB, heap released: 29 MiB), 129 MiB/134 MiB CGO alloc/total (209.7 CGO/sec), 99.6/0.2 %(u/s)time, 0.0 %gc (197x), 416 B/416 B (r/w)net W240504 07:07:32.597908 10331 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=default,rpc] 1681 latency jump (prev avg 69.47ms, current 151.48ms) I240504 07:07:34.097537 8926 2@util/log/event_log.go:32 [T1,Vsystem,n1] 1682 ={"Timestamp":1714806454097531003,"EventType":"runtime_stats","MemRSSBytes":1165959168,"GoroutineCount":304,"MemStackSysBytes":3473408,"GoAllocBytes":716619488,"GoTotalBytes":868772312,"HeapFragmentBytes":53698848,"HeapReservedBytes":68222976,"HeapReleasedBytes":30285824,"CGoAllocBytes":134925632,"CGoTotalBytes":140947456,"CGoCallRate":259.8702,"CPUUserPercent":99.75824,"CPUSysPercent":0.19951646,"GCPausePercent":0.00047756263,"GCRunCount":198,"NetHostRecvBytes":416,"NetHostSendBytes":416} I240504 07:07:34.097736 8926 2@server/status/runtime_log.go:47 [T1,Vsystem,n1] 1683 runtime stats: 1.1 GiB RSS, 304 goroutines (stacks: 3.3 MiB), 683 MiB/828 MiB Go alloc/total (heap fragmentation: 51 MiB, heap reserved: 65 MiB, heap released: 29 MiB), 129 MiB/134 MiB CGO alloc/total (259.9 CGO/sec), 99.8/0.2 %(u/s)time, 0.0 %gc (198x), 416 B/416 B (r/w)net W240504 07:07:35.828900 10344 2@rpc/clock_offset.go:291 [T1,Vsystem,n1,rnode=1,raddr=127.0.0.1:33781,class=system,rpc] 1684 latency jump (prev avg 74.81ms, current 123.46ms) ```
Parameters:
attempt=1run=5shard=4Help
See also: How To Investigate a Go Test Failure (internal)
/cc @cockroachdb/release-engThis test on roachdash | Improve this report!
Jira issue: CRDB-38428