Closed cockroach-teamcity closed 2 years ago
roachtest.autoupgrade failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:
test artifacts and logs in: /artifacts/autoupgrade/run_1
autoupgrade.go:188,autoupgrade.go:262,test_runner.go:896: pq: result is ambiguous: replica unavailable: (n5,s5):4 unable to serve request to r47:/Table/{6-8} [(n5,s5):4, (n4,s4):5, (n2,s2):3, next=6, gen=18]: lost quorum (down: (n4,s4):5,(n2,s2):3); closed timestamp: 1661235039.101590008,0 (2022-08-23 06:10:39); raft status: {"id":"4","term":11,"vote":"4","commit":50,"lead":"4","raftState":"StateLeader","applied":50,"progress":{"3":{"match":50,"next":51,"state":"StateProbe"},"4":{"match":93,"next":94,"state":"StateReplicate"},"5":{"match":0,"next":37,"state":"StateProbe"}},"leadtransferee":"0"}: have been waiting 61.80s for slow proposal Put [/Table/6/1/"cluster.preserve_downgrade_option"/0,/Min), [txn: 5ff399ca], [can-forward-ts]
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 003c0360de8b64319b5f0f127b99be91dbdca8a3:
| 3 true 35 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 32 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 31 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 29 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 27 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 25 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 23 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 21 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 20 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 19 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 17 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 16 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 15 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 13 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 11 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 9 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 7 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 6 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 5 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 4 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 3 true decommissioning false
| id is_live replicas is_decommissioning membership is_draining
| 3 true 2 true decommissioning false
Wraps: (4) COMMAND_PROBLEM
Wraps: (5) Node 3. Command with error:
| ``````
| ./cockroach node decommission 3 --insecure --port={pgport:3}
| ``````
Wraps: (6) exit status 1
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
Quick update on this:
GCE_PROJECT=andrei-jepsen ./pkg/cmd/roachtest/roachstress.sh -uc100 -b autoupgrade -- --cpu-quota 1300 --debug
passed with no failures. I'll spin up more runs now but I suspect some of our recent decommissioning changes (https://github.com/cockroachdb/cockroach/pull/85640 and https://github.com/cockroachdb/cockroach/pull/86701) have made this roachtest faster and perhaps hidden the regression that was surfaced in the above failures.
roachtest.autoupgrade failed with artifacts on master @ a82711442c65cf14489c55041b45b11a1e38415b:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1823,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:906: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1777
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1822
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:906
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 389661e823c19f318fa07ec2278336262531692d:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ bc2e47da0523b347c28cf024707e80cd35d6c98a:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
This test has started reliably failing with nodes that crash with the log message:
E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 ⋮ [-] 19 ‹ERROR›: cockroach server exited with error: store ‹node1,node1store1,store1=/mnt/data1/cockroach›, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)
@dt would probably have thoughts here.
roachtest.autoupgrade failed with artifacts on master @ 2cfa6b1779ead01508c04e228bf72b4e0e96d98c:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 95677eb5f8d006629b16024fb7d87d55344c1470:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 1f24e52aea4195ffe26b97160079bb29f264338a:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ bd97ad5b8c9f537a89492a051574d867469bef33:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 3fa1a1600898d7b78b9e39d07132a387a2f9a1b6:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
roachtest.autoupgrade failed with artifacts on master @ 87ed064dc23eab6948ee8a07e8507f150bda0e44:
test artifacts and logs in: /artifacts/autoupgrade/run_1
cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
| github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
| github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
| github.com/cockroachdb/cockroach/pkg/roachprod.Start
| github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
| main.(*clusterImpl).StartE
| main/pkg/cmd/roachtest/cluster.go:1814
| main.(*clusterImpl).Start
| main/pkg/cmd/roachtest/cluster.go:1859
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
| github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
| main.(*testRunner).runTest.func2
| main/pkg/cmd/roachtest/test_runner.go:917
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) one or more parallel execution failure
Error types: (1) *withstack.withStack (2) *errutil.leafError
Parameters: ROACHTEST_cloud=gce
, ROACHTEST_cpu=4
, ROACHTEST_ssd=0
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
Reassigning this to @dt for the most recent failure mode:
E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 ⋮ [-] 19 ‹ERROR›: cockroach server exited with error: store ‹node1,node1store1,store1=/mnt/data1/cockroach›, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)
Perhaps a result of #86345?
Yeah, seems related to that work.
I think @srosenberg has a blanket fix for these in #88005
also the version thing isn't a ga or release blocker since it only affects master.
I think the GA blocker here was for the original failure, before the version upgrades started failing. Re-labeling, but @nvanbenschoten can feel free to remove otherwise.
Closing, as we have not seen the original failure mode in over a month and Aayush was unable to reproduce in https://github.com/cockroachdb/cockroach/issues/86520#issuecomment-1238257351.
roachtest.autoupgrade failed with artifacts on master @ e6a7dc2f8ee39549e186bd05626c4c375b76fd04:
Parameters:
ROACHTEST_cloud=gce
,ROACHTEST_cpu=4
,ROACHTEST_ssd=0
Help
See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)
/cc @cockroachdb/kv-triage
This test on roachdash | Improve this report!
Jira issue: CRDB-18792
Epic CRDB-19172