Closed cockroach-teamcity closed 3 years ago
See #66944 for details.
There was an attempted fix for this in #67916, but seems like it didn't take.
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 701b177d8f4b81d8654dfb4090a2cd3cf82e63a7:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[1:{3,6,7} 4:{3,7,8} 5:{3,7,8} 7:{3,6,8} 8:{3,7,8} 17:{3,6,7} 35:{3,6,7} 49:{3,6,7} 52:{3,6,8} 56:{3,7,8} 63:{3,6,8} 66:{3,6,7} 74:{3,7,8} 77:{3,6,7} 79:{3,6,7} 90:{3,7,8} 104:{3,6,8} 106:{3,6,8} 115:{3,7,8} 117:{3,6,7} 121:{3,6,8} 138:{3,6,8} 151:{3,6,8} 154:{3,6,7} 155:{3,6,8}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[1:{3,6,7} 4:{3,7,8} 5:{3,7,8} 7:{3,6,8} 8:{3,7,8} 17:{3,6,7} 35:{3,6,7} 49:{3,6,7} 52:{3,6,8} 56:{3,7,8} 63:{3,6,8} 66:{3,6,7} 74:{3,7,8} 77:{3,6,7} 79:{3,6,7} 90:{3,7,8} 104:{3,6,8} 106:{3,6,8} 115:{3,7,8} 117:{3,6,7} 121:{3,6,8} 138:{3,6,8} 151:{3,6,8} 154:{3,6,7} 155:{3,6,8}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3246869-1627712255-68-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 6: 10937
4: 10773
3: dead (exit status 137)
1: 11722
2: 10501
5: 11157
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 31af9e32a55a166166e9ba9c5327b7cd847ae236:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[17:{3,5,6} 23:{3,4,6} 37:{3,4,6} 43:{3,5,6} 54:{3,4,6} 60:{3,4,6} 62:{3,5,6} 81:{3,4,5} 97:{3,4,6} 101:{3,4,6} 111:{3,4,6} 118:{3,5,6} 120:{3,4,6} 131:{3,4,5} 133:{3,4,5} 135:{3,5,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[17:{3,5,6} 23:{3,4,6} 37:{3,4,6} 43:{3,5,6} 54:{3,4,6} 60:{3,4,6} 62:{3,5,6} 81:{3,4,5} 97:{3,4,6} 101:{3,4,6} 111:{3,4,6} 118:{3,5,6} 120:{3,4,6} 131:{3,4,5} 133:{3,4,5} 135:{3,5,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3248647-1627798575-67-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 6: 10877
5: 10647
3: dead (exit status 137)
4: 10700
2: 11419
1: 11263
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ c995342ead51e08f8ed1155de4218d30a00d86d2:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[2:{3,4,5} 22:{3,4,5} 40:{3,5,6} 44:{3,4,5} 100:{3,5,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[2:{3,4,5} 22:{3,4,5} 40:{3,5,6} 44:{3,4,5} 100:{3,5,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3250445-1627885444-69-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: dead (exit status 137)
6: 10662
5: 10721
4: 10664
2: 10628
1: 11074
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 1c46e1cd4e5be986bf9d13799bb7e13ddc896ed2:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[1:{3,4,5} 7:{3,5,6} 10:{3,4,6} 13:{3,5,6} 15:{3,5,6} 25:{3,4,6} 26:{3,4,6} 31:{3,5,6} 32:{3,5,6} 33:{3,4,5} 35:{3,4,6} 37:{3,5,6} 50:{3,4,5} 51:{3,5,6} 54:{3,4,6} 72:{3,5,6} 76:{3,4,6} 80:{3,5,6} 90:{3,4,6} 96:{3,4,6} 123:{3,4,5} 130:{3,4,5} 131:{3,4,6} 139:{3,5,6} 140:{3,4,5} 142:{3,4,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[1:{3,4,5} 7:{3,5,6} 10:{3,4,6} 13:{3,5,6} 15:{3,5,6} 25:{3,4,6} 26:{3,4,6} 31:{3,5,6} 32:{3,5,6} 33:{3,4,5} 35:{3,4,6} 37:{3,5,6} 50:{3,4,5} 51:{3,5,6} 54:{3,4,6} 72:{3,5,6} 76:{3,4,6} 80:{3,5,6} 90:{3,4,6} 96:{3,4,6} 123:{3,4,5} 130:{3,4,5} 131:{3,4,6} 139:{3,5,6} 140:{3,4,5} 142:{3,4,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3254514-1627971576-62-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: dead (exit status 137)
4: 11082
2: 10898
1: 11978
5: 11084
6: 11071
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ eef03a46f2e43ff70485dadf7d9ad445db05cab4:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[9:{3,4,6} 14:{3,4,5} 19:{3,5,6} 26:{3,5,6} 28:{3,4,6} 29:{3,4,6} 31:{3,4,6} 34:{3,5,6} 39:{3,4,5} 64:{3,5,6} 65:{3,4,5} 85:{3,4,5} 88:{3,4,5} 95:{3,5,6} 98:{3,5,6} 111:{3,4,6} 112:{3,4,5} 122:{3,4,6} 133:{3,5,6} 135:{3,4,5} 136:{3,4,5} 140:{3,5,6} 153:{3,5,6} 159:{3,4,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[9:{3,4,6} 14:{3,4,5} 19:{3,5,6} 26:{3,5,6} 28:{3,4,6} 29:{3,4,6} 31:{3,4,6} 34:{3,5,6} 39:{3,4,5} 64:{3,5,6} 65:{3,4,5} 85:{3,4,5} 88:{3,4,5} 95:{3,5,6} 98:{3,5,6} 111:{3,4,6} 112:{3,4,5} 122:{3,4,6} 133:{3,5,6} 135:{3,4,5} 136:{3,4,5} 140:{3,5,6} 153:{3,5,6} 159:{3,4,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3259029-1628057587-61-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 6: 11403
1: 11443
5: 11375
3: dead (exit status 137)
2: 11582
4: 11808
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 6b8d59327add74cf1342345fb3eaffc3a3e765d2:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[4:{3,6,8} 15:{3,7,8} 19:{3,7,8} 39:{3,6,8} 42:{3,7,8} 47:{3,6,7} 58:{3,6,7} 64:{3,6,7} 65:{3,7,8} 67:{3,6,8} 68:{3,6,7} 72:{3,6,7} 74:{3,7,8} 80:{3,6,7} 82:{3,7,8} 86:{3,6,8} 87:{3,6,7} 94:{3,6,8} 97:{3,7,8} 98:{3,6,7} 105:{3,6,8} 106:{3,6,7} 113:{3,6,8} 117:{3,6,7} 124:{3,6,7} 126:{3,6,8} 133:{3,7,8} 135:{3,6,8} 144:{3,6,7} 146:{3,6,7} 154:{3,6,7}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[4:{3,6,8} 15:{3,7,8} 19:{3,7,8} 39:{3,6,8} 42:{3,7,8} 47:{3,6,7} 58:{3,6,7} 64:{3,6,7} 65:{3,7,8} 67:{3,6,8} 68:{3,6,7} 72:{3,6,7} 74:{3,7,8} 80:{3,6,7} 82:{3,7,8} 86:{3,6,8} 87:{3,6,7} 94:{3,6,8} 97:{3,7,8} 98:{3,6,7} 105:{3,6,8} 106:{3,6,7} 113:{3,6,8} 117:{3,6,7} 124:{3,6,7} 126:{3,6,8} 133:{3,7,8} 135:{3,6,8} 144:{3,6,7} 146:{3,6,7} 154:{3,6,7}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3264559-1628144240-61-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 11538
5: 11366
2: 11881
4: 12305
3: dead (exit status 137)
6: 11463
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 50ef2fc205baa65c5a740c2d614fe1de279367e9:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[8:{3,6,8} 9:{3,6,8} 10:{3,6,7} 16:{3,6,8} 18:{3,6,7} 24:{3,7,8} 28:{3,7,8} 31:{3,6,7} 32:{3,6,7} 35:{3,6,8} 41:{3,7,8} 53:{3,7,8} 57:{3,7,8} 58:{3,6,7} 60:{3,6,8} 62:{3,6,7} 67:{3,6,8} 68:{3,6,7} 72:{3,6,7} 75:{3,6,7} 76:{3,6,7} 77:{3,7,8} 80:{3,6,7} 86:{3,6,8} 88:{3,6,8} 99:{3,7,8} 102:{3,6,8} 103:{3,6,7} 109:{3,6,7} 115:{3,7,8} 120:{3,7,8} 121:{3,7,8} 123:{3,6,8} 133:{3,6,8} 137:{3,7,8}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[8:{3,6,8} 9:{3,6,8} 10:{3,6,7} 16:{3,6,8} 18:{3,6,7} 24:{3,7,8} 28:{3,7,8} 31:{3,6,7} 32:{3,6,7} 35:{3,6,8} 41:{3,7,8} 53:{3,7,8} 57:{3,7,8} 58:{3,6,7} 60:{3,6,8} 62:{3,6,7} 67:{3,6,8} 68:{3,6,7} 72:{3,6,7} 75:{3,6,7} 76:{3,6,7} 77:{3,7,8} 80:{3,6,7} 86:{3,6,8} 88:{3,6,8} 99:{3,7,8} 102:{3,6,8} 103:{3,6,7} 109:{3,6,7} 115:{3,7,8} 120:{3,7,8} 121:{3,7,8} 123:{3,6,8} 133:{3,6,8} 137:{3,7,8}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3269276-1628230445-62-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 11718
6: 11567
2: 11901
3: dead (exit status 137)
4: 11709
5: 11635
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ cab185ff71f0924953d987fe6ffd14efdd32a3a0:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[1:{3,5,6} 16:{3,4,5} 18:{3,4,5} 19:{3,5,6} 22:{3,5,6} 24:{3,4,5} 34:{3,5,6} 36:{3,4,6} 37:{3,4,6} 39:{3,4,6} 41:{3,4,5} 46:{3,4,6} 47:{3,5,6} 50:{3,5,6} 51:{3,5,6} 53:{3,4,5} 54:{3,5,6} 58:{3,5,6} 62:{3,4,5} 68:{3,5,6} 70:{3,4,5} 75:{3,4,5} 76:{3,5,6} 77:{3,4,6} 80:{3,4,6} 84:{3,5,6} 92:{3,5,6} 93:{3,5,6} 95:{3,4,5} 100:{3,4,6} 101:{3,4,6} 103:{3,4,6} 104:{3,4,5} 107:{3,5,6} 108:{3,5,6} 111:{3,5,6} 112:{3,5,6} 115:{3,4,6} 116:{3,4,6} 119:{3,5,6} 120:{3,5,6} 123:{3,4,6} 136:{3,4,6} 137:{3,4,6} 138:{3,4,5} 141:{3,4,5}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[1:{3,5,6} 16:{3,4,5} 18:{3,4,5} 19:{3,5,6} 22:{3,5,6} 24:{3,4,5} 34:{3,5,6} 36:{3,4,6} 37:{3,4,6} 39:{3,4,6} 41:{3,4,5} 46:{3,4,6} 47:{3,5,6} 50:{3,5,6} 51:{3,5,6} 53:{3,4,5} 54:{3,5,6} 58:{3,5,6} 62:{3,4,5} 68:{3,5,6} 70:{3,4,5} 75:{3,4,5} 76:{3,5,6} 77:{3,4,6} 80:{3,4,6} 84:{3,5,6} 92:{3,5,6} 93:{3,5,6} 95:{3,4,5} 100:{3,4,6} 101:{3,4,6} 103:{3,4,6} 104:{3,4,5} 107:{3,5,6} 108:{3,5,6} 111:{3,5,6} 112:{3,5,6} 115:{3,4,6} 116:{3,4,6} 119:{3,5,6} 120:{3,5,6} 123:{3,4,6} 136:{3,4,6} 137:{3,4,6} 138:{3,4,5} 141:{3,4,5}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3273416-1628317362-65-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 1: 12287
4: 12163
3: dead (exit status 137)
2: 11923
6: 12283
5: 12059
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 62ec88c61edcaa023a579199cc5b43d3ee951cef:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[12:{3,5,6} 19:{3,4,6} 34:{3,5,6} 40:{3,5,6} 43:{3,5,6} 45:{3,4,6} 58:{3,4,5} 65:{3,4,6} 72:{3,5,6} 78:{3,5,6} 92:{3,4,5} 95:{3,5,6} 110:{3,4,5} 113:{3,5,6} 118:{3,4,5} 119:{3,4,6} 121:{3,5,6} 123:{3,5,6} 129:{3,4,5} 130:{3,5,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[12:{3,5,6} 19:{3,4,6} 34:{3,5,6} 40:{3,5,6} 43:{3,5,6} 45:{3,4,6} 58:{3,4,5} 65:{3,4,6} 72:{3,5,6} 78:{3,5,6} 92:{3,4,5} 95:{3,5,6} 110:{3,4,5} 113:{3,5,6} 118:{3,4,5} 119:{3,4,6} 121:{3,5,6} 123:{3,5,6} 129:{3,4,5} 130:{3,5,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3275637-1628489663-60-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: dead (exit status 137)
2: 11867
5: 11818
4: 11745
1: 12371
6: 11883
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 847514dab6354d4cc4ccf7b2857487b32119fb37:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[5:{3,6,8} 8:{3,6,7} 13:{3,7,8} 22:{3,7,8} 33:{3,6,8} 38:{3,7,8} 46:{3,6,7} 50:{3,6,8} 51:{3,6,8} 56:{3,6,8} 58:{3,6,8} 62:{3,6,8} 67:{3,7,8} 68:{3,6,8} 71:{3,6,7} 79:{3,6,8} 83:{3,6,8} 85:{3,6,8} 91:{3,7,8} 92:{3,6,8} 96:{3,7,8} 108:{3,6,7} 116:{3,6,7} 117:{3,7,8} 118:{3,6,8} 120:{3,6,7} 122:{3,7,8} 127:{3,6,7} 135:{3,6,7}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[5:{3,6,8} 8:{3,6,7} 13:{3,7,8} 22:{3,7,8} 33:{3,6,8} 38:{3,7,8} 46:{3,6,7} 50:{3,6,8} 51:{3,6,8} 56:{3,6,8} 58:{3,6,8} 62:{3,6,8} 67:{3,7,8} 68:{3,6,8} 71:{3,6,7} 79:{3,6,8} 83:{3,6,8} 85:{3,6,8} 91:{3,7,8} 92:{3,6,8} 96:{3,7,8} 108:{3,6,7} 116:{3,6,7} 117:{3,7,8} 118:{3,6,8} 120:{3,6,7} 122:{3,7,8} 127:{3,6,7} 135:{3,6,7}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3281225-1628577374-61-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 12124
6: 12279
1: 12224
3: dead (exit status 137)
5: 11976
4: 11516
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1171
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:279
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2094
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 3928f1963833fbf51ae47bd2a42ae6a200ebbb14:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[8:{3,4,6} 15:{3,5,6} 19:{3,5,6} 29:{3,4,5} 35:{3,4,5} 41:{3,5,6} 46:{3,4,5} 48:{3,4,5} 67:{3,4,5} 75:{3,4,5} 79:{3,4,5} 81:{3,4,6} 84:{3,5,6} 87:{3,4,5} 88:{3,5,6} 89:{3,4,5} 92:{3,5,6} 100:{3,4,5} 101:{3,5,6} 106:{3,4,5} 115:{3,5,6} 118:{3,4,5} 121:{3,4,6} 126:{3,4,6} 127:{3,4,5} 134:{3,4,6} 135:{3,4,5} 153:{3,4,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[8:{3,4,6} 15:{3,5,6} 19:{3,5,6} 29:{3,4,5} 35:{3,4,5} 41:{3,5,6} 46:{3,4,5} 48:{3,4,5} 67:{3,4,5} 75:{3,4,5} 79:{3,4,5} 81:{3,4,6} 84:{3,5,6} 87:{3,4,5} 88:{3,5,6} 89:{3,4,5} 92:{3,5,6} 100:{3,4,5} 101:{3,5,6} 106:{3,4,5} 115:{3,5,6} 118:{3,4,5} 121:{3,4,6} 126:{3,4,6} 127:{3,4,5} 134:{3,4,6} 135:{3,4,5} 153:{3,4,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3296712-1628835955-63-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 3: dead (exit status 137)
5: 11804
1: 11858
4: 11889
6: 11979
2: 12923
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 0880e83e30ee5eb9aab7bb2297324e098d028225:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[7:{3,5,6} 9:{3,4,5} 11:{3,4,5} 16:{3,4,5} 25:{3,4,6} 27:{3,5,6} 37:{3,5,6} 48:{3,5,6} 52:{3,4,5} 56:{3,5,6} 57:{3,4,6} 60:{3,5,6} 62:{3,5,6} 63:{3,5,6} 86:{3,5,6} 98:{3,4,5} 103:{3,4,6} 108:{3,4,5} 127:{3,4,5} 133:{3,4,6} 157:{3,4,5}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[7:{3,5,6} 9:{3,4,5} 11:{3,4,5} 16:{3,4,5} 25:{3,4,6} 27:{3,5,6} 37:{3,5,6} 48:{3,5,6} 52:{3,4,5} 56:{3,5,6} 57:{3,4,6} 60:{3,5,6} 62:{3,5,6} 63:{3,5,6} 86:{3,5,6} 98:{3,4,5} 103:{3,4,6} 108:{3,4,5} 127:{3,4,5} 133:{3,4,6} 157:{3,4,5}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3301517-1628922680-62-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 6: 11966
1: 11701
3: dead (exit status 137)
5: 12309
2: 11957
4: 11392
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ ee3efd6b1e24a3e1676778f5028fa0a35266f683:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[2:{3,4,5} 3:{3,4,6} 6:{3,5,6} 12:{3,4,6} 13:{3,4,6} 15:{3,4,5} 16:{3,4,5} 30:{3,4,6} 37:{3,4,6} 43:{3,5,6} 55:{3,4,5} 57:{3,4,6} 59:{3,4,6} 69:{3,4,5} 75:{3,5,6} 85:{3,4,6} 95:{3,4,6} 107:{3,4,5} 111:{3,5,6} 116:{3,4,5} 122:{3,5,6} 142:{3,4,5}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[2:{3,4,5} 3:{3,4,6} 6:{3,5,6} 12:{3,4,6} 13:{3,4,6} 15:{3,4,5} 16:{3,4,5} 30:{3,4,6} 37:{3,4,6} 43:{3,5,6} 55:{3,4,5} 57:{3,4,6} 59:{3,4,6} 69:{3,4,5} 75:{3,5,6} 85:{3,4,6} 95:{3,4,6} 107:{3,4,5} 111:{3,5,6} 116:{3,4,5} 122:{3,5,6} 142:{3,4,5}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3303319-1629008692-61-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 6: 11777
3: dead (exit status 137)
5: 12125
1: 12403
2: 12129
4: 12227
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ 04a41e7915f4a89dcc1d0dbd92466c6adf79ec9f:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[3:{3,4,5} 13:{3,4,6} 15:{3,4,6} 19:{3,4,6} 21:{3,4,6} 23:{3,5,6} 29:{3,5,6} 30:{3,5,6} 39:{3,5,6} 40:{3,4,5} 41:{3,5,6} 47:{3,5,6} 52:{3,5,6} 55:{3,4,5} 60:{3,4,6} 63:{3,4,6} 67:{3,5,6} 68:{3,4,6} 73:{3,4,6} 79:{3,4,5} 81:{3,4,6} 83:{3,4,6} 84:{3,4,6} 88:{3,4,5} 95:{3,5,6} 97:{3,4,6} 110:{3,5,6} 112:{3,4,5} 116:{3,4,5} 117:{3,5,6} 118:{3,5,6} 119:{3,4,5} 124:{3,4,5} 127:{3,4,6} 133:{3,4,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[3:{3,4,5} 13:{3,4,6} 15:{3,4,6} 19:{3,4,6} 21:{3,4,6} 23:{3,5,6} 29:{3,5,6} 30:{3,5,6} 39:{3,5,6} 40:{3,4,5} 41:{3,5,6} 47:{3,5,6} 52:{3,5,6} 55:{3,4,5} 60:{3,4,6} 63:{3,4,6} 67:{3,5,6} 68:{3,4,6} 73:{3,4,6} 79:{3,4,5} 81:{3,4,6} 83:{3,4,6} 84:{3,4,6} 88:{3,4,5} 95:{3,5,6} 97:{3,4,6} 110:{3,5,6} 112:{3,4,5} 116:{3,4,5} 117:{3,5,6} 118:{3,5,6} 119:{3,4,5} 124:{3,4,5} 127:{3,4,6} 133:{3,4,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:867: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3305136-1629094127-63-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 2: 11679
5: 11828
3: dead (exit status 137)
1: 12262
6: 11734
4: 11941
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ dd82053908203cf6d77c36c06a8280831bb93d57:
The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/replicagc-changed-peers/restart=true/run_1
replicagc.go:288,replicagc.go:102,replicagc.go:36,test_runner.go:777: ranges remained on n3 (according to meta2): map[31:{3,5,6} 39:{3,5,6} 47:{3,5,6} 67:{3,4,6} 88:{3,5,6} 97:{3,5,6} 102:{3,4,6} 128:{3,4,6} 144:{3,5,6}]
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:286
| github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:197
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForZeroReplicasOnN3
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:265
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runReplicaGCChangedPeers
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:102
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerReplicaGC.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/replicagc.go:36
| main.(*testRunner).runTest.func2
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachtest/test_runner.go:777
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (2) ranges remained on n3 (according to meta2): map[31:{3,5,6} 39:{3,5,6} 47:{3,5,6} 67:{3,4,6} 88:{3,5,6} 97:{3,5,6} 102:{3,4,6} 128:{3,4,6} 144:{3,5,6}]
Error types: (1) *withstack.withStack (2) *errutil.leafError
cluster.go:1245,context.go:89,cluster.go:1233,test_runner.go:866: dead node detection: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod monitor teamcity-3311212-1629181178-65-n6cpu4 --oneshot --ignore-empty-nodes: exit status 1 4: 11410
6: 12151
3: dead (exit status 137)
2: 11949
1: 11990
5: 12220
Error: UNCLASSIFIED_PROBLEM: 3: dead (exit status 137)
(1) UNCLASSIFIED_PROBLEM
Wraps: (2) attached stack trace
-- stack trace:
| main.glob..func14
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:1173
| main.wrap.func1
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:281
| github.com/spf13/cobra.(*Command).execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:856
| github.com/spf13/cobra.(*Command).ExecuteC
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:960
| github.com/spf13/cobra.(*Command).Execute
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/vendor/github.com/spf13/cobra/command.go:897
| main.main
| /home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/main.go:2107
| runtime.main
| /usr/local/go/src/runtime/proc.go:225
| runtime.goexit
| /usr/local/go/src/runtime/asm_amd64.s:1371
Wraps: (3) 3: dead (exit status 137)
Error types: (1) errors.Unclassified (2) *withstack.withStack (3) *errutil.leafError
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest) See: [CI job to stress roachtests](https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_Nightlies_RoachtestStress)
For the CI stress job, click the ellipsis (...) next to the Run button and fill in: * Changes / Build branch: master * Parameters / `env.TESTS`: `^replicagc-changed-peers/restart=true$` * Parameters / `env.COUNT`: <number of runs>
/cc @cockroachdb/kv-triage
Going to run a roachstress build (https://teamcity.cockroachdb.com/viewQueued.html?itemId=3313313&tab=queuedBuildOverviewTab) on https://github.com/cockroachdb/cockroach/pull/67916 itself. That landed 22 days ago, but the failures here and in the restart=false sibling only started 18 days ago (plus I tested the original commit and did not see this issue). Perhaps roachtest didn't run, perhaps something else broke this test. The failure in this test is that replicas don't move off n3 within five minutes, which is unexpected to me; n3 is decommissioning and the node is not running, but there are five other nodes around. The replicas that get stuck on n3 are all vanilla table ranges.
That didn't work since the roachstress script the job uses wasn't checked in yet, I'll try with the PR that introduces it instead: https://teamcity.cockroachdb.com/viewLog.html?buildId=3313386&buildTypeId=Cockroach_Nightlies_RoachtestStress
Didn't work too:
stderr: ERROR: (gcloud.compute.instances.list) Some requests did not succeed:
- Required 'compute.instances.list' permission for 'projects/andrei-jepsen'
Error: UNCLASSIFIED_PROBLEM: failed to run: gcloud compute instances list --project andrei-jepsen --format json: exit status 1
(1) UNCLASSIFIED_PROBLEM
``
I'll have to use cockroach-ephemeral I guess.
Will put this on ice for now, and use `roachstress.sh` when I get back to this.
Must've changed the service account permissions since I set the stress job up. I opened a dev-inf ticket to get a new project for these: https://cockroachlabs.atlassian.net/browse/DEVINF-140.
Long fixed on master (which is this branch)
roachtest.replicagc-changed-peers/restart=true failed with artifacts on master @ fe1fb73ae989142193643db30dd4b1b6dd6fe7dd:
Reproduce
See: [roachtest README](https://github.com/cockroachdb/cockroach/tree/master/pkg/cmd/roachtest)
/cc @cockroachdb/kv-triage
This test on roachdash | Improve this report!