search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

roachtest: cluster_creation failed #112176

Open cockroach-teamcity opened 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test tpccbench/nodes=3/cpu=16/mt-shared-process was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-32275

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test sysbench/oltp_insert/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:698).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,cluster=teamcity-12137068-1697002307-147-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-11t20_21_25z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1939509501 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12137068-1697002307-147-n9cpu8-geo-0004 teamcity-12137068-1697002307-147-n9cpu8-geo-0005 teamcity-12137068-1697002307-147-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test sysbench/oltp_read_only/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test sysbench/oltp_update_index/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 437fa1b407bb5078a2a98c0fb5b8e5436a3be926:

test sysbench/oltp_write_only/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ fe7abe63ff6ac18b59971c5c6e21f43f4e8b5f5a:

test tpccbench/nodes=3/cpu=16/mt-shared-process was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ fe7abe63ff6ac18b59971c5c6e21f43f4e8b5f5a:

test sysbench/oltp_insert/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test schemachange/bulkingest was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test tpccbench/nodes=3/cpu=16/lease=expiration was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ bd71370a4afb6cf2a7718a4f20ccba6a9436bdb6:

test loqrecovery/workload=tpcc/rangeSize=16mb was skipped due to (test_runner.go:698).func3: cluster.PutE: put /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 failed: ~ scp -r -C -o StrictHostKeyChecking=no -o ConnectTimeout=10 -i /home/roach/.ssh/id_rsa -i /home/roach/.ssh/google_compute_engine /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ubuntu@104.196.164.162:./cockroach-default
ssh: connect to host 104.196.164.162 port 22: Connection timed out
lost connection: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7) See: [Grafana](https://go.crdb.dev/roachtest-grafana/teamcity-12162093/loqrecovery-workload-tpcc-rangesize-16mb/-62135596800000/-62135596680000)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test tpccbench/nodes=3/cpu=16/mt-shared-process was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test sysbench/oltp_insert/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:698).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12176641-1697261618-153-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-14t19_55_43z --metadata-from-file startup-script=/tmp/gce-startup-script3467074699 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12176641-1697261618-153-n9cpu8-geo-0001 teamcity-12176641-1697261618-153-n9cpu8-geo-0002 teamcity-12176641-1697261618-153-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test ycsb/A/nodes=3/cpu=32 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ abd56c6d6e9dbf131fc1fa0d3f8d1fbcfb7e4901:

test ycsb/B/nodes=3/cpu=32 was skipped due to (test_runner.go:698).func3: no allocation specified; cfg.alloc must be set

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ c5dc7554124b2de67f9db3e5f37b82037859f5b7:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-BtOrAL2qDvJdQqDoOrH_Ri5gmZc= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-18T09:24:42Z},{Key=Name,Value=teamcity-12229630-1697607208-35-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12229630-1697607208-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-18T09:24:42Z},{Key=Name,Value=teamcity-12229630-1697607208-35-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12229630-1697607208-35-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script2878060607 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2868326610 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ c5dc7554124b2de67f9db3e5f37b82037859f5b7:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-BtOrAL2qDvJdQqDoOrH_Ri5gmZc= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-18T09:26:58Z},{Key=Name,Value=teamcity-12229630-1697607208-38-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12229630-1697607208-38-n4cpu96},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-18T09:26:58Z},{Key=Name,Value=teamcity-12229630-1697607208-38-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12229630-1697607208-38-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1192227289 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2850139650 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ c5dc7554124b2de67f9db3e5f37b82037859f5b7:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12229629-1697607215-162-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-18t19_53_14z --metadata-from-file startup-script=/tmp/gce-startup-script3147324240 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12229629-1697607215-162-n9cpu8-geo-0001 teamcity-12229629-1697607215-162-n9cpu8-geo-0002 teamcity-12229629-1697607215-162-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ c5dc7554124b2de67f9db3e5f37b82037859f5b7:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230919'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-d, us-east1-c
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-d,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 3744e5655559d3350aedaf1fbbaecbf1e69c98d6:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-d2KtfivgAQyaTxEDp9lZoQfZTc8= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-19T09:21:49Z},{Key=Name,Value=teamcity-12247720-1697693542-38-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12247720-1697693542-38-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-19T09:21:49Z},{Key=Name,Value=teamcity-12247720-1697693542-38-n4cpu96-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12247720-1697693542-38-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script1508674448 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping935541866 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 3744e5655559d3350aedaf1fbbaecbf1e69c98d6:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-d2KtfivgAQyaTxEDp9lZoQfZTc8= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-19T09:24:02Z},{Key=Name,Value=teamcity-12247720-1697693542-41-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12247720-1697693542-41-n4cpu96},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-19T09:24:02Z},{Key=Name,Value=teamcity-12247720-1697693542-41-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12247720-1697693542-41-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script736592741 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping262903245 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 3744e5655559d3350aedaf1fbbaecbf1e69c98d6:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-10-19t19_43_23z,roachprod=true,cluster=teamcity-12247719-1697693585-166-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script2817413870 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12247719-1697693585-166-n9cpu8-geo-0007 teamcity-12247719-1697693585-166-n9cpu8-geo-0008 teamcity-12247719-1697693585-166-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 935399600693f84fb746715f430ecc245934f3cd:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-C2BXX-M80R5ADq9_MU6PlMS4WZA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12265150-1697779958-37-n4cpu96-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12265150-1697779958-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-20T09:29:10Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-12265150-1697779958-37-n4cpu96-0004},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12265150-1697779958-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-20T09:29:10Z}] --user-data file:///tmp/aws-startup-script3463991101 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2346750280 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 935399600693f84fb746715f430ecc245934f3cd:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-C2BXX-M80R5ADq9_MU6PlMS4WZA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-20T09:31:49Z},{Key=Name,Value=teamcity-12265150-1697779958-40-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12265150-1697779958-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2023-10-20T09:31:49Z},{Key=Name,Value=teamcity-12265150-1697779958-40-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12265150-1697779958-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script2015494951 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3466343578 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 935399600693f84fb746715f430ecc245934f3cd:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12271771-1697813609-151-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-21t04_34_31z --metadata-from-file startup-script=/tmp/gce-startup-script3548148987 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12271771-1697813609-151-n9cpu8-geo-0004 teamcity-12271771-1697813609-151-n9cpu8-geo-0005 teamcity-12271771-1697813609-151-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12281325-1697866340-156-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-21t19_50_00z --metadata-from-file startup-script=/tmp/gce-startup-script2760522646 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12281325-1697866340-156-n9cpu8-geo-0001 teamcity-12281325-1697866340-156-n9cpu8-geo-0002 teamcity-12281325-1697866340-156-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-sAnfaytbvVCSnC-SfQ9RqjPfZo4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292159-1697952633-38-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T09:23:18Z},{Key=Name,Value=teamcity-12292159-1697952633-38-n4cpu96-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292159-1697952633-38-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T09:23:18Z},{Key=Name,Value=teamcity-12292159-1697952633-38-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script1152080391 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2687462397 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-sAnfaytbvVCSnC-SfQ9RqjPfZo4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T09:25:39Z},{Key=Name,Value=teamcity-12292159-1697952633-41-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292159-1697952633-41-n4cpu96}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-22T09:25:39Z},{Key=Name,Value=teamcity-12292159-1697952633-41-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12292159-1697952633-41-n4cpu96}] --user-data file:///tmp/aws-startup-script11076768 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2836051939 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-10-22t20_12_49z,roachprod=true,cluster=teamcity-12292158-1697952684-158-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script1240208100 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12292158-1697952684-158-n9cpu8-geo-0001 teamcity-12292158-1697952684-158-n9cpu8-geo-0002 teamcity-12292158-1697952684-158-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-oDfaTnyai27R17WmXrjYrH9qSoM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12302442-1698039158-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-23T09:10:52Z},{Key=Name,Value=teamcity-12302442-1698039158-36-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12302442-1698039158-36-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-23T09:10:52Z},{Key=Name,Value=teamcity-12302442-1698039158-36-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script2166786374 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2257012417 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-oDfaTnyai27R17WmXrjYrH9qSoM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12302442-1698039158-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-23T09:13:05Z},{Key=Name,Value=teamcity-12302442-1698039158-39-n4cpu96-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12302442-1698039158-39-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-23T09:13:05Z},{Key=Name,Value=teamcity-12302442-1698039158-39-n4cpu96-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1401064234 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3832197210 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ dc24db6b7ed2f53d034ae9b8038ce08fc7d57542:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-10-23t20_02_43z,roachprod=true,cluster=teamcity-12302441-1698039099-157-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script2635481053 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12302441-1698039099-157-n9cpu8-geo-0004 teamcity-12302441-1698039099-157-n9cpu8-geo-0005 teamcity-12302441-1698039099-157-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ af2cdcea915f024eae56aee7b3dee8794cbc44e1:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2023-10-24t19_36_48z,roachprod=true,cluster=teamcity-12320973-1698125502-157-n9cpu8-geo --metadata-from-file startup-script=/tmp/gce-startup-script125910542 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12320973-1698125502-157-n9cpu8-geo-0001 teamcity-12320973-1698125502-157-n9cpu8-geo-0002 teamcity-12320973-1698125502-157-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ d3d723a0f49fb16ccfca5ddf6521b7c1376c808b:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-TWekgIijzt3-cbNaaLN76BL9kiA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340181-1698211925-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T09:04:34Z},{Key=Name,Value=teamcity-12340181-1698211925-37-n4cpu96-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340181-1698211925-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T09:04:34Z},{Key=Name,Value=teamcity-12340181-1698211925-37-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script4168534523 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1318961744 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ d3d723a0f49fb16ccfca5ddf6521b7c1376c808b:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-TWekgIijzt3-cbNaaLN76BL9kiA= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340181-1698211925-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T09:06:44Z},{Key=Name,Value=teamcity-12340181-1698211925-40-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12340181-1698211925-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-25T09:06:44Z},{Key=Name,Value=teamcity-12340181-1698211925-40-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script92494643 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1830730947 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ d3d723a0f49fb16ccfca5ddf6521b7c1376c808b:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,created=2023-10-25t20_13_04z,roachprod=true,cluster=teamcity-12340180-1698211934-164-n9cpu8-geo,lifetime=12h0m0s,arch=amd64 --metadata-from-file startup-script=/tmp/gce-startup-script768390314 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12340180-1698211934-164-n9cpu8-geo-0007 teamcity-12340180-1698211934-164-n9cpu8-geo-0008 teamcity-12340180-1698211934-164-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ ff055942cbd72f699764a4c7d185af79d878e036:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-HnVu93S_13me4GgKoDzuB0BXOEI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-26T05:42:40Z},{Key=Name,Value=teamcity-12359449-1698298349-12-n2cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12359449-1698298349-12-n2cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-26T05:42:40Z},{Key=Name,Value=teamcity-12359449-1698298349-12-n2cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12359449-1698298349-12-n2cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script30615585 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping654312139 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ ff055942cbd72f699764a4c7d185af79d878e036:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-HnVu93S_13me4GgKoDzuB0BXOEI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12359449-1698298349-21-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-26T05:51:00Z},{Key=Name,Value=teamcity-12359449-1698298349-21-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12359449-1698298349-21-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-26T05:51:00Z},{Key=Name,Value=teamcity-12359449-1698298349-21-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script2034779964 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1541850103 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ ff055942cbd72f699764a4c7d185af79d878e036:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12359448-1698298259-158-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-26t19_55_16z --metadata-from-file startup-script=/tmp/gce-startup-script1550775726 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12359448-1698298259-158-n9cpu8-geo-0001 teamcity-12359448-1698298259-158-n9cpu8-geo-0002 teamcity-12359448-1698298259-158-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 5e8db8ab50ba3695684e29a5e6f94e1dbaa573b1:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-0RWK2MsKkZH8_UVyuyyIyuYAp6g= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12375827-1698384687-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-27T09:04:25Z},{Key=Name,Value=teamcity-12375827-1698384687-37-n4cpu96-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-12375827-1698384687-37-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-27T09:04:25Z},{Key=Name,Value=teamcity-12375827-1698384687-37-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script234766476 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2158402816 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 5e8db8ab50ba3695684e29a5e6f94e1dbaa573b1:

test kv95/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:703).func3: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-0RWK2MsKkZH8_UVyuyyIyuYAp6g= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e3d146e87ebc5a2c --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12375827-1698384687-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-27T09:06:40Z},{Key=Name,Value=teamcity-12375827-1698384687-40-n4cpu96-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-12375827-1698384687-40-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-10-27T09:06:40Z},{Key=Name,Value=teamcity-12375827-1698384687-40-n4cpu96-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2812303439 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1354110113 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c6id.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 5e8db8ab50ba3695684e29a5e6f94e1dbaa573b1:

test costfuzz/seed-multi-region was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --labels usage=roachtest,roachprod=true,cluster=teamcity-12375826-1698384735-65-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-27t13_24_36z --metadata-from-file startup-script=/tmp/gce-startup-script3558802331 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-c teamcity-12375826-1698384735-65-n9cpu4-geo-0005]
Output: WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231025a'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-c zone. Consider trying your request in the us-west1-b, us-west1-a
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-c
      zonesAvailable: us-west1-b,us-west1-a
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=4 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 5e8db8ab50ba3695684e29a5e6f94e1dbaa573b1:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12375826-1698384735-162-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-27t20_12_27z --metadata-from-file startup-script=/tmp/gce-startup-script2685502632 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12375826-1698384735-162-n9cpu8-geo-0001 teamcity-12375826-1698384735-162-n9cpu8-geo-0002 teamcity-12375826-1698384735-162-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 877e24c91113ef9fe67fae44511f9f77cf2650c2:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12391479-1698471170-158-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-28t20_16_11z --metadata-from-file startup-script=/tmp/gce-startup-script3448484523 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-12391479-1698471170-158-n9cpu8-geo-0007 teamcity-12391479-1698471170-158-n9cpu8-geo-0008 teamcity-12391479-1698471170-158-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 6aa847e0e6f7eb6ab26dd9b150f165ccae9dd2c6:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12401722-1698557512-160-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-29t19_59_05z --metadata-from-file startup-script=/tmp/gce-startup-script4063206848 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west-2b teamcity-12401722-1698557512-160-n9cpu8-geo-0004 teamcity-12401722-1698557512-160-n9cpu8-geo-0005 teamcity-12401722-1698557512-160-n9cpu8-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-west-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 1e5be546f219b64b171cb1f39985706b5809d79b:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,arch=amd64,created=2023-10-30t19_54_47z,roachprod=true,cluster=teamcity-12412122-1698643872-162-n9cpu8-geo,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4114521566 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12412122-1698643872-162-n9cpu8-geo-0001 teamcity-12412122-1698643872-162-n9cpu8-geo-0002 teamcity-12412122-1698643872-162-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 348d11425a1184ce4de8a6f8dc85995cd3c653bc:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:703).func3: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-12428970-1698730390-172-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-10-31t20_09_34z --metadata-from-file startup-script=/tmp/gce-startup-script207612878 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-12428970-1698730390-172-n9cpu8-geo-0001 teamcity-12428970-1698730390-172-n9cpu8-geo-0002 teamcity-12428970-1698730390-172-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_arch=amd64 , ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.2 @ 348d11425a1184ce4de8a6f8dc85995cd3c653bc:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20231030'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highcpu-96 VM instance with 16 local SSD(s) is currently unavailable
      in the us-east1-b zone. Consider trying your request in the us-east1-c, us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2-highcpu-96
      zone: us-east1-b
      zonesAvailable: us-east1-c,us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #112528 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.12-rc.FROZEN] - #111127 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #110990 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng TODO-deprecate.branch-release-23.1.9-rc X-infra-flake] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!