cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachtest: cluster_creation failed #123745

Open cockroach-teamcity opened 6 months ago

cockroach-teamcity commented 6 months ago

roachtest.cluster_creation failed with artifacts on master @ e90d15c9807775a1a1d89f75cac4c9383c325709:

test ycsb/C/nodes=3/cpu=32 failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15141267-1715060891-170-n4cpu32-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-38523

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test perturbation/metamorphic/addNode failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test perturbation/metamorphic/decommission failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test follower-reads/mixed-version/survival=region/locality=global/reads=strong failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-09-03t12_10_45z,roachprod=true,cluster=teamcity-16726095-1725342640-126-n6cpu4-geo,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1837607275 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16726095-1725342640-126-n6cpu4-geo-0004 teamcity-16726095-1725342640-126-n6cpu4-geo-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-09-03t13_07_18z,roachprod=true,cluster=teamcity-16726095-1725342640-171-n9cpu4-geo --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1271713062 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16726095-1725342640-171-n9cpu4-geo-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16726095-1725342640-172-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-09-03t13_07_25z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1627681791 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16726095-1725342640-172-n9cpu4-geo-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test perturbation/metamorphic/partition failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test perturbation/metamorphic/addNode failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test restore/tpce/400GB/aws/nodes=4/cpus=16 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7g.4xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-Te_i1KEYg0S0wz2MBL8F6s42dE0= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16743678-1725428970-57-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-09-04T07:22:10Z},{Key=Name,Value=teamcity-16743678-1725428970-57-n4cpu16-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16743678-1725428970-57-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-09-04T07:22:10Z},{Key=Name,Value=teamcity-16743678-1725428970-57-n4cpu16-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2964038653 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3288615691 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7g.4xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7g.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-Te_i1KEYg0S0wz2MBL8F6s42dE0= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16743678-1725428970-66-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16743678-1725428970-66-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-04T07:36:50Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16743678-1725428970-66-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16743678-1725428970-66-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-04T07:36:50Z}] --user-data file:///tmp/aws-startup-script2436720321 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1351291087 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test kv95/enc=false/nodes=3/tracing failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-KYR8IkQxSZqfOYOTsCvgr2quK7U= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-09-05T07:08:21Z},{Key=Name,Value=teamcity-16760065-1725515315-53-n4cpu8-0004},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16760065-1725515315-53-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-09-05T07:08:21Z},{Key=Name,Value=teamcity-16760065-1725515315-53-n4cpu8-0004},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16760065-1725515315-53-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1591896929 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1233019796 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test perturbation/metamorphic/partition failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test perturbation/metamorphic/addNode failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-09-05t09_52_23z,roachprod=true,cluster=teamcity-16760064-1725515361-86-n9cpu4-geo,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script837311276 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16760064-1725515361-86-n9cpu4-geo-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-16760064-1725515361-89-n9cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-09-05t09_59_34z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script484819201 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16760064-1725515361-89-n9cpu4-geo-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test indexes/2/nodes=6/multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-09-05t11_45_27z,roachprod=true,cluster=teamcity-16760064-1725515361-124-n7cpu16-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4146608126 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16760064-1725515361-124-n7cpu16-geo-0003 teamcity-16760064-1725515361-124-n7cpu16-geo-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ b6862f42d3a229cad814b61d5a77a420afb540b5:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-09-05t13_23_26z,roachprod=true,cluster=teamcity-16760064-1725515361-183-n9cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4261972675 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-west1-b teamcity-16760064-1725515361-183-n9cpu4-geo-0002]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-west1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-west1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-west1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-Nv8QTXusVCCXsGpPN8Ap8Sw0ovU= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-09-06T07:45:31Z},{Key=Name,Value=teamcity-16774771-1725601768-45-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16774771-1725601768-45-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-09-06T07:45:31Z},{Key=Name,Value=teamcity-16774771-1725601768-45-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16774771-1725601768-45-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script1831555794 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping732900263 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6a.24xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6a.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: in provider: gce: T2A instances are not supported outside of [asia-southeast1-b,asia-southeast1-c,europe-west4-a,europe-west4-b,europe-west4-c,us-central1-a,us-central1-b,us-central1-f] [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.cluster_creation failed with artifacts on master @ 950a090603bffa9216f01b03aeeb7dd093c6be64:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130222 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130221 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-23.2] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #130219 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130218 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130217 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130216 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130215 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130214 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #130213 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!