roachtest: cluster_creation failed

cockroach-teamcity commented 4 months ago

roachtest.cluster_creation failed with artifacts on master @ e90d15c9807775a1a1d89f75cac4c9383c325709:

test ycsb/C/nodes=3/cpu=32 failed: (test_runner.go:791).func4: in provider: azure: creating VM teamcity-15141267-1715060891-170-n4cpu32-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPLDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 350, Current Usage: 320, Additional Required: 32, (Minimum) New Limit Required: 352. Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPLDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:352,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPLDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!

Jira issue: CRDB-38523}

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ dafb6dd507b38fb3d6eb8b7e2493c7b8abed34d2:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ dafb6dd507b38fb3d6eb8b7e2493c7b8abed34d2:

      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ dafb6dd507b38fb3d6eb8b7e2493c7b8abed34d2:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-08-30t11_49_34z,roachprod=true,cluster=teamcity-16680691-1724996973-179-n4cpu32,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-custom-32-65536 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script86451375 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-16680691-1724996973-179-n4cpu32-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-custom-32-65536 VM instance with 4 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:4
      vmType: n2-custom-32-65536
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ dafb6dd507b38fb3d6eb8b7e2493c7b8abed34d2:

      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highmem-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-highmem-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-highmem-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test kv0/enc=false/nodes=3/cpu=32 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-3t5ONz4KDUhlJ6Sfq-6ur1-bXtI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-37-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:43:14Z},{Key=Name,Value=teamcity-16695536-1725083458-37-n4cpu32-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-37-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:43:14Z},{Key=Name,Value=teamcity-16695536-1725083458-37-n4cpu32-0002}] --user-data file:///tmp/aws-startup-script603877893 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3322927002 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test kv95/enc=false/nodes=1/cpu=32 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-3t5ONz4KDUhlJ6Sfq-6ur1-bXtI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16695536-1725083458-45-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-45-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:54:54Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-16695536-1725083458-45-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-45-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:54:54Z}] --user-data file:///tmp/aws-startup-script404164203 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4226264385 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-3t5ONz4KDUhlJ6Sfq-6ur1-bXtI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:59:49Z},{Key=Name,Value=teamcity-16695536-1725083458-49-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-49-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-08-31T06:59:49Z},{Key=Name,Value=teamcity-16695536-1725083458-49-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16695536-1725083458-49-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1375513412 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3239332647 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/addNode failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=2
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test cdc/ledger failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,created=2024-08-31t08_13_28z,roachprod=true,cluster=teamcity-16695535-1725083473-63-n4cpu16,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4036560325 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-16695535-1725083473-63-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/addNode failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-qaA0vBeGTZ0KMF4Pg3fcaB5EyiM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16705039-1725169703-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-01T07:39:21Z},{Key=Name,Value=teamcity-16705039-1725169703-41-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16705039-1725169703-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-01T07:39:21Z},{Key=Name,Value=teamcity-16705039-1725169703-41-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1039152307 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2219263721 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-qaA0vBeGTZ0KMF4Pg3fcaB5EyiM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16705039-1725169703-46-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-01T07:45:06Z},{Key=Name,Value=teamcity-16705039-1725169703-46-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16705039-1725169703-46-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-01T07:45:06Z},{Key=Name,Value=teamcity-16705039-1725169703-46-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script558929960 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2964980993 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/decommission failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/restart failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/partition failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on azure [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=2
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test perturbation/metamorphic/decommission failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test cdc/pubsub-sink failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16714036-1725256176-73-n4cpu16,lifetime=12h0m0s,arch=amd64,created=2024-09-02t07_49_37z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1952636954 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-16714036-1725256176-73-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test failover/non-system/blackhole-send failed: (test_runner.go:841).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-09-02t10_34_08z,roachprod=true,cluster=teamcity-16714036-1725256176-128-n7cpu2 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2389276134 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-16714036-1725256176-128-n7cpu2-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test schemachange/random-load failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar issue without the B-runtime-assertions-enabled label. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=true
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=all/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

test unoptimized-query-oracle/disable-rules=half/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-16 VM instance with 2 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:2
      vmType: n2-standard-16
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-2 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-2
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2-standard-4 VM instance with 1 local SSD(s) is currently unavailable
      in the us-east1-b zone. Try requesting the VM in another zone. For more information,
      view the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:1
      vmType: n2-standard-4
      zone: us-east1-b
      zonesAvailable: ''
    reason: stockout
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test perturbation/metamorphic/partition failed: (test_runner.go:841).func4: specifying SSD count is not yet supported on aws [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=1
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test ycsb/C/nodes=3/cpu=32 failed: (test_runner.go:841).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-JlmQDnPkOwB3kLkq0tpdCu4Danw= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16726096-1725342587-65-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-03T07:38:15Z},{Key=Name,Value=teamcity-16726096-1725342587-65-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-16726096-1725342587-65-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-03T07:38:15Z},{Key=Name,Value=teamcity-16726096-1725342587-65-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1162114516 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2132161648 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test costfuzz/rand-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on master @ 4142920c2d5c50c0520c124764aeeda94ba043ae:

test costfuzz/seed-multi-region failed: (test_runner.go:841).func4: one or more parallel execution failure(s): COMMAND_PROBLEM: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #128956 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.10-rc] - #128942 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.4-rc] - #128531 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.25-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126641 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.24-rc] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #123745