search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

roachtest: cluster_creation failed #130746

Closed cockroach-teamcity closed 1 month ago

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ 174a8953ca6cb442f9b90be89cbf083df8af1036:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-09-14t11_43_39z,roachprod=true,cluster=teamcity-16884084-1726293048-104-n5cpu96,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script365564461 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-16884084-1726293048-104-n5cpu96-0001 teamcity-16884084-1726293048-104-n5cpu96-0002 teamcity-16884084-1726293048-104-n5cpu96-0003 teamcity-16884084-1726293048-104-n5cpu96-0004 teamcity-16884084-1726293048-104-n5cpu96-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-16884084-1726293048-104-n5cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-16884084-1726293048-104-n5cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240904'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-42203

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ 9612ab97923c20e80388cbe5bd7c6997ffde62bf:

test connection_latency/nodes=3/certs failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-16894193-1726379512-32-n4cpu4,lifetime=12h0m0s,arch=arm64,created=2024-09-15t08_03_41z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script2920975617 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-b teamcity-16894193-1726379512-32-n4cpu4-0001 teamcity-16894193-1726379512-32-n4cpu4-0002 teamcity-16894193-1726379512-32-n4cpu4-0003 teamcity-16894193-1726379512-32-n4cpu4-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-b/instances/teamcity-16894193-1726379512-32-n4cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-b/instances/teamcity-16894193-1726379512-32-n4cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240904'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ bf1ecaa86c48f31a0cfcc8e42b23fc856736554e:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:758).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-yuW7GS-NcAx1Ak0jjoVw-5bBwsU= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16918922-1726552402-09-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-17T06:02:58Z},{Key=Name,Value=teamcity-16918922-1726552402-09-n4cpu96-0003}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-16918922-1726552402-09-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-17T06:02:58Z},{Key=Name,Value=teamcity-16918922-1726552402-09-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script3262487143 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping912734287 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6a.24xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6a.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #126293 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.7-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ c3ddfa7a1b062e6829c74dbf4067ef0479092b36:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:758).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-2vjLS5FsPBO1C2mskP8NoGFfrh8= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-26T06:18:28Z},{Key=Name,Value=teamcity-17043856-1727329975-21-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17043856-1727329975-21-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-26T06:18:28Z},{Key=Name,Value=teamcity-17043856-1727329975-21-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17043856-1727329975-21-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script333666828 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping924517567 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131334 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.27-rc] - #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ c3ddfa7a1b062e6829c74dbf4067ef0479092b36:

test kv95/enc=false/nodes=3/cpu=32/splt=0 failed: (test_runner.go:758).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-2vjLS5FsPBO1C2mskP8NoGFfrh8= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17043856-1727329975-25-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-26T06:21:59Z},{Key=Name,Value=teamcity-17043856-1727329975-25-n4cpu32-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17043856-1727329975-25-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-26T06:21:59Z},{Key=Name,Value=teamcity-17043856-1727329975-25-n4cpu32-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script160681110 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3971459117 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131334 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.27-rc] - #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ c3ddfa7a1b062e6829c74dbf4067ef0479092b36:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-17076262-1727502703-33-n6cpu16,lifetime=12h0m0s,arch=arm64,created=2024-09-28t08_45_40z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3348503766 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17076262-1727502703-33-n6cpu16-0001 teamcity-17076262-1727502703-33-n6cpu16-0002 teamcity-17076262-1727502703-33-n6cpu16-0003 teamcity-17076262-1727502703-33-n6cpu16-0004 teamcity-17076262-1727502703-33-n6cpu16-0005 teamcity-17076262-1727502703-33-n6cpu16-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17076262-1727502703-33-n6cpu16-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131334 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.27-rc] - #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ c3ddfa7a1b062e6829c74dbf4067ef0479092b36:

test cdc/scan/catchup-cold/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=rangefeed/format=json/sink=null failed: (test_runner.go:758).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17085619-1727589211-33-n6cpu16,lifetime=12h0m0s,arch=arm64,created=2024-09-29t07_45_24z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script4273581650 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17085619-1727589211-33-n6cpu16-0001 teamcity-17085619-1727589211-33-n6cpu16-0002 teamcity-17085619-1727589211-33-n6cpu16-0003 teamcity-17085619-1727589211-33-n6cpu16-0004 teamcity-17085619-1727589211-33-n6cpu16-0005 teamcity-17085619-1727589211-33-n6cpu16-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17085619-1727589211-33-n6cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17085619-1727589211-33-n6cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17085619-1727589211-33-n6cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17085619-1727589211-33-n6cpu16-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131334 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.27-rc] - #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-23.2.12-rc @ c3ddfa7a1b062e6829c74dbf4067ef0479092b36:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:758).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-2ZS14PvQ5avH9gBnoIpa4uOh13s= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17094451-1727675680-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-30T06:08:16Z},{Key=Name,Value=teamcity-17094451-1727675680-12-n4cpu96-0002},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17094451-1727675680-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-09-30T06:08:16Z},{Key=Name,Value=teamcity-17094451-1727675680-12-n4cpu96-0002},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script19808777 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1763825163 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6a.24xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6a.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131334 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.27-rc] - #130636 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.3-rc] - #130557 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.11-rc] - #130550 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.5-rc] - #130220 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #129712 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.26-rc] - #129301 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.1-rc] - #127332 roachtest: cluster_creation failed [O-roachtest O-robot O-rsg T-testeng X-infra-flake branch-release-24.1.3-rc] - #127326 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.9-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

renatolabs commented 1 month ago

Branch deleted, closing.