search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.06k stars 3.8k forks source link

roachtest: cluster_creation failed #132974

Open cockroach-teamcity opened 5 days ago

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test admission-control/disk-bandwidth-limiter failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:14:34Z},{Key=Name,Value=teamcity-17376626-1729317016-19-n2cpu8-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:14:34Z},{Key=Name,Value=teamcity-17376626-1729317016-19-n2cpu8-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script863458723 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1321626091 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43373

cockroach-teamcity commented 1 day ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ f4bf28879d49979d438967379b44c2fe3464f968:

 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-c zone. Consider trying your request in the us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-c
      zonesAvailable: us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-c zone. Consider trying your request in the us-east1-d
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-c
      zonesAvailable: us-east1-d
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test cdc/tpcc-1000/sink=kafka failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-30-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:29:37Z},{Key=Name,Value=teamcity-17434700-1729749107-30-n4cpu16-0004},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-30-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:29:37Z},{Key=Name,Value=teamcity-17434700-1729749107-30-n4cpu16-0004},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1626250195 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1783772936 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6id.xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m6id.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=1/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-34-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:32:28Z},{Key=Name,Value=teamcity-17434700-1729749107-34-n2cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-34-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:32:28Z},{Key=Name,Value=teamcity-17434700-1729749107-34-n2cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1493481393 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping479373626 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-37-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:33:56Z},{Key=Name,Value=teamcity-17434700-1729749107-37-n2cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-37-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:33:56Z},{Key=Name,Value=teamcity-17434700-1729749107-37-n2cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2001810126 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3077770144 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-44-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:40:50Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-44-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:40:50Z}] --user-data file:///tmp/aws-startup-script715421772 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2623775984 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-47-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:42:28Z},{Key=Name,Value=teamcity-17434700-1729749107-47-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-47-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:42:28Z},{Key=Name,Value=teamcity-17434700-1729749107-47-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script343751772 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1608206731 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:44:05Z},{Key=Name,Value=teamcity-17434700-1729749107-50-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:44:05Z},{Key=Name,Value=teamcity-17434700-1729749107-50-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1626169307 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping497536433 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:45:42Z},{Key=Name,Value=teamcity-17434700-1729749107-53-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-53-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:45:42Z},{Key=Name,Value=teamcity-17434700-1729749107-53-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-53-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2945845681 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping790427476 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-62-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:56:02Z},{Key=Name,Value=teamcity-17434700-1729749107-62-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-62-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T06:56:02Z},{Key=Name,Value=teamcity-17434700-1729749107-62-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3597789604 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4266337574 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv0/enc=true/nodes=1 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-65-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:57:29Z},{Key=Name,Value=teamcity-17434700-1729749107-65-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-65-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T06:57:29Z},{Key=Name,Value=teamcity-17434700-1729749107-65-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1859798892 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1227590761 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv95/enc=false/nodes=1/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T07:01:55Z},{Key=Name,Value=teamcity-17434700-1729749107-70-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-70-n2cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T07:01:55Z},{Key=Name,Value=teamcity-17434700-1729749107-70-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-70-n2cpu32}] --user-data file:///tmp/aws-startup-script3861047723 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1425853089 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-24T07:04:01Z},{Key=Name,Value=teamcity-17434700-1729749107-73-n2cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-73-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-24T07:04:01Z},{Key=Name,Value=teamcity-17434700-1729749107-73-n2cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-73-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script4212534585 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2657142405 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 20 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv95/enc=false/nodes=3/batch=16 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-79-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-79-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T07:09:26Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-79-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-79-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T07:09:26Z}] --user-data file:///tmp/aws-startup-script1071471220 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping876402129 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv95/enc=true/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.2xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-91-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T07:32:36Z},{Key=Name,Value=teamcity-17434700-1729749107-91-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-91-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T07:32:36Z},{Key=Name,Value=teamcity-17434700-1729749107-91-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2379311743 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping291764504 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test kv95limited-spans/enc=false/nodes=1/seq/no-load-splitting failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-95-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-95-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T07:35:47Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-95-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-95-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T07:35:47Z}] --user-data file:///tmp/aws-startup-script3333976216 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3605339232 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/A/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.2xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-121-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T08:09:28Z},{Key=Name,Value=teamcity-17434700-1729749107-121-n4cpu8-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-121-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T08:09:28Z},{Key=Name,Value=teamcity-17434700-1729749107-121-n4cpu8-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script759458514 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3275115529 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6id.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m6id.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/E/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-24T08:20:36Z},{Key=Name,Value=teamcity-17434700-1729749107-131-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-131-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-24T08:20:36Z},{Key=Name,Value=teamcity-17434700-1729749107-131-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-131-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script2252048042 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping315980627 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 18 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-137-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-137-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T08:27:04Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-137-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-137-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T08:27:04Z}] --user-data file:///tmp/aws-startup-script1969888624 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3120110597 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 18 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/A/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-140-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T08:28:50Z},{Key=Name,Value=teamcity-17434700-1729749107-140-n4cpu32-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-140-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T08:28:50Z},{Key=Name,Value=teamcity-17434700-1729749107-140-n4cpu32-0003}] --user-data file:///tmp/aws-startup-script2750453571 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3663806659 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 18 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-143-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T08:30:34Z},{Key=Name,Value=teamcity-17434700-1729749107-143-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-143-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T08:30:34Z},{Key=Name,Value=teamcity-17434700-1729749107-143-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1069957431 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3726952697 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/C/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:03:46Z},{Key=Name,Value=teamcity-17434700-1729749107-149-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-149-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:03:46Z},{Key=Name,Value=teamcity-17434700-1729749107-149-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-149-n4cpu32}] --user-data file:///tmp/aws-startup-script503398534 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2888363874 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-152-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:05:35Z},{Key=Name,Value=teamcity-17434700-1729749107-152-n4cpu32-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-152-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:05:35Z},{Key=Name,Value=teamcity-17434700-1729749107-152-n4cpu32-0003}] --user-data file:///tmp/aws-startup-script3528199696 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping701316255 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/D/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-155-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:07:14Z},{Key=Name,Value=teamcity-17434700-1729749107-155-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-155-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:07:14Z},{Key=Name,Value=teamcity-17434700-1729749107-155-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script798910022 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping851157795 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/E/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-158-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-158-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T10:08:55Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17434700-1729749107-158-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-158-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T10:08:55Z}] --user-data file:///tmp/aws-startup-script612218915 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3351496497 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/E/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-161-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T10:10:33Z},{Key=Name,Value=teamcity-17434700-1729749107-161-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17434700-1729749107-161-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-24T10:10:33Z},{Key=Name,Value=teamcity-17434700-1729749107-161-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script318547324 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3544779306 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/F/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-165-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:13:33Z},{Key=Name,Value=teamcity-17434700-1729749107-165-n4cpu32-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-165-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:13:33Z},{Key=Name,Value=teamcity-17434700-1729749107-165-n4cpu32-0002}] --user-data file:///tmp/aws-startup-script1341387587 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2668715650 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test ycsb/F/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-lxb9hZuWY-I7Pbo5Fv_xyi9tFUI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:15:10Z},{Key=Name,Value=teamcity-17434700-1729749107-168-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-168-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-24T10:15:10Z},{Key=Name,Value=teamcity-17434700-1729749107-168-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17434700-1729749107-168-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2369299887 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2059557220 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c7gd.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c7gd.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 16 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

test costfuzz/rand-multi-region failed: (test_runner.go:846).func4: in provider: gce: T2A instances are not supported outside of [asia-southeast1-b,asia-southeast1-c,europe-west4-a,europe-west4-b,europe-west4-c,us-central1-a,us-central1-b,us-central1-f] [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 11 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-b zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-b
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-b zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-b
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 hours ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-c zone. Consider trying your request in the us-east1-b
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-c
      zonesAvailable: us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-c zone. Consider trying your request in the us-east1-b
      zone(s), which currently has capacity to accommodate your request. Alternatively,
      you can try your request again with a different VM hardware configuration or
      at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-c
      zonesAvailable: us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-c' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!