roachtest: autoupgrade failed

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ e6a7dc2f8ee39549e186bd05626c4c375b76fd04:

          | 3   true    43  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    41  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    38  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    35  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    32  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    29  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    27  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    26  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    24  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    23  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    21  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    19  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    18  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    15  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    13  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    12  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    7   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    6   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    5   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    4   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    3   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    2   true    decommissioning false
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 3. Command with error:
          | ``````
          | ./cockroach node decommission 3 --insecure --port={pgport:3}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-18792

Epic CRDB-19172

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 80c274877a917580af62be6eb0cd48c8c7ae9c08:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    autoupgrade.go:188,autoupgrade.go:262,test_runner.go:896: pq: result is ambiguous: replica unavailable: (n5,s5):4 unable to serve request to r47:/Table/{6-8} [(n5,s5):4, (n4,s4):5, (n2,s2):3, next=6, gen=18]: lost quorum (down: (n4,s4):5,(n2,s2):3); closed timestamp: 1661235039.101590008,0 (2022-08-23 06:10:39); raft status: {"id":"4","term":11,"vote":"4","commit":50,"lead":"4","raftState":"StateLeader","applied":50,"progress":{"3":{"match":50,"next":51,"state":"StateProbe"},"4":{"match":93,"next":94,"state":"StateReplicate"},"5":{"match":0,"next":37,"state":"StateProbe"}},"leadtransferee":"0"}: have been waiting 61.80s for slow proposal Put [/Table/6/1/"cluster.preserve_downgrade_option"/0,/Min), [txn: 5ff399ca], [can-forward-ts]

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 003c0360de8b64319b5f0f127b99be91dbdca8a3:

          | 3   true    35  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    32  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    31  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    29  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    27  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    25  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    23  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    21  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    20  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    19  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    17  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    16  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    15  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    13  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    11  true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    9   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    7   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    6   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    5   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    4   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    3   true    decommissioning false
          | id  is_live replicas    is_decommissioning  membership  is_draining
          | 3   true    2   true    decommissioning false
        Wraps: (4) COMMAND_PROBLEM
        Wraps: (5) Node 3. Command with error:
          | ``````
          | ./cockroach node decommission 3 --insecure --port={pgport:3}
          | ``````
        Wraps: (6) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) errors.Cmd (5) *hintdetail.withDetail (6) *exec.ExitError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[aayushshah15]

Quick update on this:

GCE_PROJECT=andrei-jepsen ./pkg/cmd/roachtest/roachstress.sh -uc100 -b autoupgrade -- --cpu-quota 1300 --debug

passed with no failures. I'll spin up more runs now but I suspect some of our recent decommissioning changes (https://github.com/cockroachdb/cockroach/pull/85640 and https://github.com/cockroachdb/cockroach/pull/86701) have made this roachtest faster and perhaps hidden the regression that was surfaced in the above failures.

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ a82711442c65cf14489c55041b45b11a1e38415b:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1823,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:906: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1777
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1822
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:906
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 389661e823c19f318fa07ec2278336262531692d:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ bc2e47da0523b347c28cf024707e80cd35d6c98a:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 773568fbda06ba9be9fb1bc34a331f21c8891ffa:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[nvanbenschoten]

This test has started reliably failing with nodes that crash with the log message:

E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 ⋮ [-] 19  ‹ERROR›: cockroach server exited with error: store ‹node1,node1store1,store1=/mnt/data1/cockroach›, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)

[erikgrinaker]

@dt would probably have thoughts here.

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 2cfa6b1779ead01508c04e228bf72b4e0e96d98c:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 95677eb5f8d006629b16024fb7d87d55344c1470:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 1f24e52aea4195ffe26b97160079bb29f264338a:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 9a05046ce19e7678340e82c70d61e928be95bc72:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ bd97ad5b8c9f537a89492a051574d867469bef33:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 3fa1a1600898d7b78b9e39d07132a387a2f9a1b6:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.autoupgrade failed with artifacts on master @ 87ed064dc23eab6948ee8a07e8507f150bda0e44:

test artifacts and logs in: /artifacts/autoupgrade/run_1
    cluster.go:1860,autoupgrade.go:129,autoupgrade.go:262,test_runner.go:917: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2286
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2167
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:155
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:662
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1814
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1859
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func1
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:129
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerAutoUpgrade.func2
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/autoupgrade.go:262
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:917
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1594
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

[erikgrinaker]

Reassigning this to @dt for the most recent failure mode:

E220909 06:03:14.180386 1 1@cli/clierror/check.go:35 ⋮ [-] 19  ‹ERROR›: cockroach server exited with error: store ‹node1,node1store1,store1=/mnt/data1/cockroach›, last used with cockroach version v22.1, is too old for running version v1000022.1-70 (which requires data from v1000022.1 or later)

[irfansharif]

Perhaps a result of #86345?

[erikgrinaker]

Yeah, seems related to that work.

[dt]

I think @srosenberg has a blanket fix for these in #88005

[dt]

also the version thing isn't a ga or release blocker since it only affects master.

[erikgrinaker]

I think the GA blocker here was for the original failure, before the version upgrades started failing. Re-labeling, but @nvanbenschoten can feel free to remove otherwise.

[nvanbenschoten]

Closing, as we have not seen the original failure mode in over a month and Aayush was unable to reproduce in https://github.com/cockroachdb/cockroach/issues/86520#issuecomment-1238257351.