search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.85k stars 3.77k forks source link

roachtest: cluster_creation failed #127326

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc] - #120620 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-40381

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc] - #120620 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=rangefeed/format=json/sink=null failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16105809-1721195380-39-n6cpu16,lifetime=12h0m0s,arch=arm64,created=2024-07-17t07_53_11z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script1218221450 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16105809-1721195380-39-n6cpu16-0001 teamcity-16105809-1721195380-39-n6cpu16-0002 teamcity-16105809-1721195380-39-n6cpu16-0003 teamcity-16105809-1721195380-39-n6cpu16-0004 teamcity-16105809-1721195380-39-n6cpu16-0005 teamcity-16105809-1721195380-39-n6cpu16-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16105809-1721195380-39-n6cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16105809-1721195380-39-n6cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16105809-1721195380-39-n6cpu16-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16105809-1721195380-39-n6cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16105809-1721195380-39-n6cpu16-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240701'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc] - #120620 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

test costfuzz/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16125800-1721281907-41-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-18t07_24_20z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script566262668 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-16125800-1721281907-41-n9cpu4-geo-0001]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240701'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Consider trying your request in the us-central1-a
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: us-central1-a
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Consider trying your request in the us-central1-a
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: us-central1-a
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c5876575d562bea45fd95ddfe6be0df69f313103:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16125800-1721281907-134-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-18t11_12_58z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3939281192 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-16125800-1721281907-134-n9cpu4-geo-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test costfuzz/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-07-19t08_34_40z,roachprod=true,cluster=teamcity-16145980-1721368283-43-n9cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3913716741 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16145980-1721368283-43-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test ledger/nodes=6/multi-az failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16145980-1721368283-104-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-19t11_06_06z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script931227292 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-c teamcity-16145980-1721368283-104-n7cpu16-geo-0005 teamcity-16145980-1721368283-104-n7cpu16-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-2
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16145980-1721368283-122-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-19t11_17_19z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1828819112 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-16145980-1721368283-122-n9cpu4-geo-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test costfuzz/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16162303-1721454728-45-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-20t09_03_50z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3549377789 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16162303-1721454728-45-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test ledger/nodes=6/multi-az failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16162303-1721454728-94-n7cpu16-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-20t11_05_51z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3369330420 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-c teamcity-16162303-1721454728-94-n7cpu16-geo-0005 teamcity-16162303-1721454728-94-n7cpu16-geo-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-c' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16162303-1721454728-109-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-20t11_23_45z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1477235773 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-16162303-1721454728-109-n9cpu4-geo-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16162303-1721454728-110-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-20t11_24_51z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1093027373 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16162303-1721454728-110-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test costfuzz/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-07-21t08_34_41z,roachprod=true,cluster=teamcity-16171861-1721541101-33-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1193628254 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16171861-1721541101-33-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test ruby-pg failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16171861-1721541101-91-n1cpu4,lifetime=12h0m0s,arch=amd64,created=2024-07-21t11_42_16z --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2517319720 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-16171861-1721541101-91-n1cpu4-0001]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ c0e26e3f66022d22c8f022652c7ce3caa57978e2:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16180423-1721627576-113-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-22t12_06_10z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script773396801 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16180423-1721627576-113-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16191007-1721713931-96-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-23t11_20_28z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2630105558 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16191007-1721713931-96-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-custom-32-65536
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test c2c/tpcc/warehouses=1000/duration=60/cutover=30 failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16199298-1721800464-27-n9cpu8,lifetime=12h0m0s,arch=arm64,created=2024-07-24t08_07_44z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type t2a-standard-8 --metadata-from-file startup-script=/tmp/gce-startup-script1834739444 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16199298-1721800464-27-n9cpu8-0001 teamcity-16199298-1721800464-27-n9cpu8-0002 teamcity-16199298-1721800464-27-n9cpu8-0003 teamcity-16199298-1721800464-27-n9cpu8-0004 teamcity-16199298-1721800464-27-n9cpu8-0005 teamcity-16199298-1721800464-27-n9cpu8-0006 teamcity-16199298-1721800464-27-n9cpu8-0007 teamcity-16199298-1721800464-27-n9cpu8-0008 teamcity-16199298-1721800464-27-n9cpu8-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16199298-1721800464-27-n9cpu8-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16199298-1721800464-27-n9cpu8-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16199298-1721800464-27-n9cpu8-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16199298-1721800464-27-n9cpu8-0008].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240720'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test costfuzz/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16199298-1721800464-43-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-24t08_47_42z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1912282997 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16199298-1721800464-43-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16199298-1721800464-90-n2cpu32,lifetime=12h0m0s,arch=amd64,created=2024-07-24t11_17_27z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-custom-32-65536 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1862727243 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-16199298-1721800464-90-n2cpu32-0001 teamcity-16199298-1721800464-90-n2cpu32-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16199298-1721800464-119-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-24t12_17_06z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1782922182 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16199298-1721800464-119-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-c zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-c
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test disk-stalled/cgroup/read-write/logs-too=false failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-07-26t09_40_16z,roachprod=true,cluster=teamcity-16217346-1721973116-45-n4cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2004-focal-arm64-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script3043648012 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16217346-1721973116-45-n4cpu4-0001 teamcity-16217346-1721973116-45-n4cpu4-0002 teamcity-16217346-1721973116-45-n4cpu4-0003 teamcity-16217346-1721973116-45-n4cpu4-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-07-26t12_09_45z,roachprod=true,cluster=teamcity-16217346-1721973116-105-n9cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script822893160 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16217346-1721973116-105-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16229872-1722059421-84-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-07-27t12_47_48z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script92056339 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16229872-1722059421-84-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-central1-f zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-central1-f
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-f' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-07-28t12_07_43z,roachprod=true,cluster=teamcity-16236821-1722145871-129-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1688685782 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-a teamcity-16236821-1722145871-129-n9cpu4-geo-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-a zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-a
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-a' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=rangefeed/format=json/sink=null failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-07-29t08_07_37z,roachprod=true,cluster=teamcity-16244085-1722232274-33-n7cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3282632193 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-16244085-1722232274-33-n7cpu16-0001 teamcity-16244085-1722232274-33-n7cpu16-0002 teamcity-16244085-1722232274-33-n7cpu16-0003 teamcity-16244085-1722232274-33-n7cpu16-0004 teamcity-16244085-1722232274-33-n7cpu16-0005 teamcity-16244085-1722232274-33-n7cpu16-0006 teamcity-16244085-1722232274-33-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16244085-1722232274-33-n7cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16244085-1722232274-33-n7cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-16244085-1722232274-33-n7cpu16-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240726'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.9-rc @ 579c0a000fb80ef7184e46ea038937c135ee2f0c:

test prune-dangling failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16244085-1722232274-38-n1cpu4,lifetime=12h0m0s,arch=amd64,created=2024-07-29t08_16_41z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1987599895 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-16244085-1722232274-38-n1cpu4-0001]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-d zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-d
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc] - #125986 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.23-rc] - #125838 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.2-rc] - #123744 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #123453 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.21-rc] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #120814 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.4-rc]

This test on roachdash | Improve this report!