roachtest: cluster_creation failed

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-41482

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test clearrange/checks=true failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-08-20t07_50_05z,roachprod=true,cluster=teamcity-16544101-1724132963-44-n10cpu16,lifetime=12h0m0s,arch=arm64 --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script470589125 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16544101-1724132963-44-n10cpu16-0001 teamcity-16544101-1724132963-44-n10cpu16-0002 teamcity-16544101-1724132963-44-n10cpu16-0003 teamcity-16544101-1724132963-44-n10cpu16-0004 teamcity-16544101-1724132963-44-n10cpu16-0005 teamcity-16544101-1724132963-44-n10cpu16-0006 teamcity-16544101-1724132963-44-n10cpu16-0007 teamcity-16544101-1724132963-44-n10cpu16-0008 teamcity-16544101-1724132963-44-n10cpu16-0009 teamcity-16544101-1724132963-44-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16544101-1724132963-44-n10cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16544101-1724132963-44-n10cpu16-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16544101-1724132963-44-n10cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16544101-1724132963-44-n10cpu16-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240801'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test sysbench/oltp_delete/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test costfuzz/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-08-22t07_39_41z,roachprod=true,cluster=teamcity-16573827-1724305721-45-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script167524081 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16573827-1724305721-45-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test costfuzz/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16588953-1724392047-48-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-23t07_59_44z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3252219941 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16588953-1724392047-48-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv95/enc=false/nodes=1/cpu=32 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16588953-1724392047-152-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-23t12_28_17z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3991998433 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16588953-1724392047-152-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-08-23t12_34_05z,roachprod=true,cluster=teamcity-16588953-1724392047-159-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3077323915 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16588953-1724392047-159-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:816).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-dJKDC1jwSrZCVRnc63VBcGLer-E= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16600474-1724478617-23-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-24T06:19:33Z},{Key=Name,Value=teamcity-16600474-1724478617-23-n4cpu32-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16600474-1724478617-23-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-24T06:19:33Z},{Key=Name,Value=teamcity-16600474-1724478617-23-n4cpu32-0002}] --user-data file:///tmp/aws-startup-script878414553 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1857959423 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:816).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-dJKDC1jwSrZCVRnc63VBcGLer-E= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16600474-1724478617-26-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16600474-1724478617-26-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-24T06:21:35Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16600474-1724478617-26-n4cpu32-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16600474-1724478617-26-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-24T06:21:35Z}] --user-data file:///tmp/aws-startup-script3455341859 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1266766881 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 23a8ffc28d2f31eef0cd13eca69693dc249d1196:

test costfuzz/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16608203-1724564863-48-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-25t07_44_55z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2850123104 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16608203-1724564863-48-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

test clock/jump/large_backward_enabled failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16614875-1724651346-40-n1cpu4,lifetime=12h0m0s,arch=arm64,created=2024-08-26t07_46_37z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy TERMINATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script3647286119 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16614875-1724651346-40-n1cpu4-0001]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-b,
      us-east1-c zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-b,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-b,
      us-east1-c zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-b,us-east1-c
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #128138 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.0-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ ae8318c64466e8a9d5908a1c99df29c0424e70be:

test clearrange/checks=true failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16628583-1724737688-53-n10cpu16,lifetime=12h0m0s,arch=arm64,created=2024-08-27t08_00_31z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script1996206586 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16628583-1724737688-53-n10cpu16-0001 teamcity-16628583-1724737688-53-n10cpu16-0002 teamcity-16628583-1724737688-53-n10cpu16-0003 teamcity-16628583-1724737688-53-n10cpu16-0004 teamcity-16628583-1724737688-53-n10cpu16-0005 teamcity-16628583-1724737688-53-n10cpu16-0006 teamcity-16628583-1724737688-53-n10cpu16-0007 teamcity-16628583-1724737688-53-n10cpu16-0008 teamcity-16628583-1724737688-53-n10cpu16-0009 teamcity-16628583-1724737688-53-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0009].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16628583-1724737688-53-n10cpu16-0010].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240801'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test costfuzz/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16645815-1724824012-45-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-28t07_54_27z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1622567056 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16645815-1724824012-45-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test follower-reads/survival=zone/locality=regional/reads=exact-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-08-28t09_07_45z,roachprod=true,cluster=teamcity-16645815-1724824012-70-n6cpu4-geo,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2098089998 --project cockroach-ephemeral --boot-disk-size=32GB --zone europe-west2-b teamcity-16645815-1724824012-70-n6cpu4-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the europe-west2-b zone. Try requesting the VM in another zone. For more
      information, view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: europe-west2-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/europe-west2-b' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16645815-1724824012-141-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-28t11_40_02z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1484141005 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16645815-1724824012-141-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16645815-1724824012-140-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-28t11_39_33z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1893969011 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16645815-1724824012-140-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 5fad41ab74c6e2f16c26553bc322fcb83fe0251d:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16645815-1724824012-147-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-28t11_46_03z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script439764745 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16645815-1724824012-147-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-34-n6cpu16,lifetime=12h0m0s,arch=arm64,created=2024-08-30t07_03_52z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3192018633 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16680811-1724996891-34-n6cpu16-0001 teamcity-16680811-1724996891-34-n6cpu16-0002 teamcity-16680811-1724996891-34-n6cpu16-0003 teamcity-16680811-1724996891-34-n6cpu16-0004 teamcity-16680811-1724996891-34-n6cpu16-0005 teamcity-16680811-1724996891-34-n6cpu16-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16680811-1724996891-34-n6cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16680811-1724996891-34-n6cpu16-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16680811-1724996891-34-n6cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16680811-1724996891-34-n6cpu16-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240829'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test costfuzz/seed-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-08-30t07_44_24z,roachprod=true,cluster=teamcity-16680811-1724996891-57-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script179972128 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16680811-1724996891-57-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test costfuzz/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-08-30t07_44_20z,roachprod=true,cluster=teamcity-16680811-1724996891-56-n9cpu4-geo,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3573463546 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16680811-1724996891-56-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/mixed-version/survival=region/locality=global/reads=strong failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-78-n6cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-30t08_35_35z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1763121755 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-78-n6cpu4-geo-0004 teamcity-16680811-1724996891-78-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/survival=region/locality=global/reads=bounded-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-81-n6cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-30t08_40_35z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1156101833 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-81-n6cpu4-geo-0004 teamcity-16680811-1724996891-81-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/survival=region/locality=regional/reads=exact-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-08-30t08_46_28z,roachprod=true,cluster=teamcity-16680811-1724996891-88-n6cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2932520675 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-88-n6cpu4-geo-0004 teamcity-16680811-1724996891-88-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/survival=zone/locality=global/reads=bounded-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-08-30t08_48_44z,roachprod=true,cluster=teamcity-16680811-1724996891-92-n6cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2159997475 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-92-n6cpu4-geo-0004 teamcity-16680811-1724996891-92-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/survival=zone/locality=global/reads=exact-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-96-n6cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-30t08_50_26z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3807296893 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-96-n6cpu4-geo-0004 teamcity-16680811-1724996891-96-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test follower-reads/survival=zone/locality=regional/reads=exact-staleness failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-99-n6cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-30t08_51_56z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1926945819 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16680811-1724996891-99-n6cpu4-geo-0004 teamcity-16680811-1724996891-99-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240829'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-d zone. Consider trying your request in the us-east1-c, us-east1-b
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-d zone. Consider trying your request in the us-east1-c, us-east1-b
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highmem-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highmem-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16680811-1724996891-164-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-08-30t10_40_54z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4206452871 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-16680811-1724996891-164-n9cpu4-geo-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-24.2.1-rc @ 41ff1f4fcbf6cd934b49c24a47a98be1f909e3e3:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #129301