search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.81k forks source link

roachtest: cluster_creation failed #135144

Open cockroach-teamcity opened 1 week ago

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:753).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-1f2BVhzMyjxIepiCUDie6AAE4KY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-11-14T07:06:25Z},{Key=Name,Value=teamcity-17753132-1731567276-10-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17753132-1731567276-10-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-11-14T07:06:25Z},{Key=Name,Value=teamcity-17753132-1731567276-10-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17753132-1731567276-10-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script540141833 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1799385632 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44372

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test kv0/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:753).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-1f2BVhzMyjxIepiCUDie6AAE4KY= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0e71591c1fe06645e --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-14T07:10:56Z},{Key=Name,Value=teamcity-17753132-1731567276-14-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17753132-1731567276-14-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-14T07:10:56Z},{Key=Name,Value=teamcity-17753132-1731567276-14-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17753132-1731567276-14-n4cpu32}] --user-data file:///tmp/aws-startup-script1056799114 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2859718365 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test c2c/shutdown/src/coordinator failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-24-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-11-14t07_25_04z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3243709621 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17753131-1731567227-24-n9cpu8-0001 teamcity-17753131-1731567227-24-n9cpu8-0002 teamcity-17753131-1731567227-24-n9cpu8-0003 teamcity-17753131-1731567227-24-n9cpu8-0004 teamcity-17753131-1731567227-24-n9cpu8-0005 teamcity-17753131-1731567227-24-n9cpu8-0006 teamcity-17753131-1731567227-24-n9cpu8-0007 teamcity-17753131-1731567227-24-n9cpu8-0008 teamcity-17753131-1731567227-24-n9cpu8-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100/protocol=mux/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-33-n6cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-14t08_06_24z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script101375170 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17753131-1731567227-33-n6cpu16-0001 teamcity-17753131-1731567227-33-n6cpu16-0002 teamcity-17753131-1731567227-33-n6cpu16-0003 teamcity-17753131-1731567227-33-n6cpu16-0004 teamcity-17753131-1731567227-33-n6cpu16-0005 teamcity-17753131-1731567227-33-n6cpu16-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=rangefeed/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t08_11_37z,roachprod=true,cluster=teamcity-17753131-1731567227-41-n7cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1516730096 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-41-n7cpu16-0001 teamcity-17753131-1731567227-41-n7cpu16-0002 teamcity-17753131-1731567227-41-n7cpu16-0003 teamcity-17753131-1731567227-41-n7cpu16-0004 teamcity-17753131-1731567227-41-n7cpu16-0005 teamcity-17753131-1731567227-41-n7cpu16-0006 teamcity-17753131-1731567227-41-n7cpu16-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-40-n7cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-14t08_11_30z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script913255198 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-40-n7cpu16-0001 teamcity-17753131-1731567227-40-n7cpu16-0002 teamcity-17753131-1731567227-40-n7cpu16-0003 teamcity-17753131-1731567227-40-n7cpu16-0004 teamcity-17753131-1731567227-40-n7cpu16-0005 teamcity-17753131-1731567227-40-n7cpu16-0006 teamcity-17753131-1731567227-40-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-40-n7cpu16-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-40-n7cpu16-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test decommission/slow failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-54-n6cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-14t09_03_13z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1144855565 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17753131-1731567227-54-n6cpu4-0001 teamcity-17753131-1731567227-54-n6cpu4-0002 teamcity-17753131-1731567227-54-n6cpu4-0003 teamcity-17753131-1731567227-54-n6cpu4-0004 teamcity-17753131-1731567227-54-n6cpu4-0005 teamcity-17753131-1731567227-54-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17753131-1731567227-54-n6cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17753131-1731567227-54-n6cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17753131-1731567227-54-n6cpu4-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-59-n5cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-14t09_07_45z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script726738917 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17753131-1731567227-59-n5cpu16-0001 teamcity-17753131-1731567227-59-n5cpu16-0002 teamcity-17753131-1731567227-59-n5cpu16-0003 teamcity-17753131-1731567227-59-n5cpu16-0004 teamcity-17753131-1731567227-59-n5cpu16-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test decommissionBench/nodes=4/warehouses=1000 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-60-n5cpu16,lifetime=12h0m0s,arch=arm64,created=2024-11-14t09_08_00z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3197726924 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17753131-1731567227-60-n5cpu16-0001 teamcity-17753131-1731567227-60-n5cpu16-0002 teamcity-17753131-1731567227-60-n5cpu16-0003 teamcity-17753131-1731567227-60-n5cpu16-0004 teamcity-17753131-1731567227-60-n5cpu16-0005]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test drain-and-decommission/nodes=9 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-14t09_11_51z,roachprod=true,cluster=teamcity-17753131-1731567227-65-n9cpu4,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3751531412 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-65-n9cpu4-0001 teamcity-17753131-1731567227-65-n9cpu4-0002 teamcity-17753131-1731567227-65-n9cpu4-0003 teamcity-17753131-1731567227-65-n9cpu4-0004 teamcity-17753131-1731567227-65-n9cpu4-0005 teamcity-17753131-1731567227-65-n9cpu4-0006 teamcity-17753131-1731567227-65-n9cpu4-0007 teamcity-17753131-1731567227-65-n9cpu4-0008 teamcity-17753131-1731567227-65-n9cpu4-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-65-n9cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-65-n9cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test drop/tpcc/w=100,nodes=9 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t09_16_50z,roachprod=true,cluster=teamcity-17753131-1731567227-70-n9cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3893863262 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17753131-1731567227-70-n9cpu4-0001 teamcity-17753131-1731567227-70-n9cpu4-0002 teamcity-17753131-1731567227-70-n9cpu4-0003 teamcity-17753131-1731567227-70-n9cpu4-0004 teamcity-17753131-1731567227-70-n9cpu4-0005 teamcity-17753131-1731567227-70-n9cpu4-0006 teamcity-17753131-1731567227-70-n9cpu4-0007 teamcity-17753131-1731567227-70-n9cpu4-0008 teamcity-17753131-1731567227-70-n9cpu4-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test clearrange/checks=false failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-14t09_17_04z,roachprod=true,cluster=teamcity-17753131-1731567227-71-n10cpu16,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3156497785 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17753131-1731567227-71-n10cpu16-0001 teamcity-17753131-1731567227-71-n10cpu16-0002 teamcity-17753131-1731567227-71-n10cpu16-0003 teamcity-17753131-1731567227-71-n10cpu16-0004 teamcity-17753131-1731567227-71-n10cpu16-0005 teamcity-17753131-1731567227-71-n10cpu16-0006 teamcity-17753131-1731567227-71-n10cpu16-0007 teamcity-17753131-1731567227-71-n10cpu16-0008 teamcity-17753131-1731567227-71-n10cpu16-0009 teamcity-17753131-1731567227-71-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-71-n10cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-71-n10cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-71-n10cpu16-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test failover/non-system/crash/lease=expiration failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17753131-1731567227-84-n7cpu2,lifetime=12h0m0s,arch=amd64,created=2024-11-14t09_40_00z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script735741382 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-84-n7cpu2-0001 teamcity-17753131-1731567227-84-n7cpu2-0002 teamcity-17753131-1731567227-84-n7cpu2-0003 teamcity-17753131-1731567227-84-n7cpu2-0004 teamcity-17753131-1731567227-84-n7cpu2-0005 teamcity-17753131-1731567227-84-n7cpu2-0006 teamcity-17753131-1731567227-84-n7cpu2-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-84-n7cpu2-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test failover/non-system/deadlock/lease=expiration failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t09_42_44z,roachprod=true,cluster=teamcity-17753131-1731567227-86-n7cpu2,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1454132763 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17753131-1731567227-86-n7cpu2-0001 teamcity-17753131-1731567227-86-n7cpu2-0002 teamcity-17753131-1731567227-86-n7cpu2-0003 teamcity-17753131-1731567227-86-n7cpu2-0004 teamcity-17753131-1731567227-86-n7cpu2-0005 teamcity-17753131-1731567227-86-n7cpu2-0006 teamcity-17753131-1731567227-86-n7cpu2-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test failover/non-system/pause failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t09_44_51z,roachprod=true,cluster=teamcity-17753131-1731567227-92-n7cpu2,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2685083602 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-92-n7cpu2-0001 teamcity-17753131-1731567227-92-n7cpu2-0002 teamcity-17753131-1731567227-92-n7cpu2-0003 teamcity-17753131-1731567227-92-n7cpu2-0004 teamcity-17753131-1731567227-92-n7cpu2-0005 teamcity-17753131-1731567227-92-n7cpu2-0006 teamcity-17753131-1731567227-92-n7cpu2-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test failover/partial/lease-gateway/lease=expiration failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,cluster=teamcity-17753131-1731567227-93-n8cpu2,lifetime=12h0m0s,arch=amd64,created=2024-11-14t09_45_28z,roachprod=true --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script986211178 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17753131-1731567227-93-n8cpu2-0001 teamcity-17753131-1731567227-93-n8cpu2-0002 teamcity-17753131-1731567227-93-n8cpu2-0003 teamcity-17753131-1731567227-93-n8cpu2-0004 teamcity-17753131-1731567227-93-n8cpu2-0005 teamcity-17753131-1731567227-93-n8cpu2-0006 teamcity-17753131-1731567227-93-n8cpu2-0007 teamcity-17753131-1731567227-93-n8cpu2-0008]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-93-n8cpu2-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-93-n8cpu2-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17753131-1731567227-93-n8cpu2-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test follower-reads/survival=zone/locality=global/reads=bounded-staleness failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t09_50_55z,roachprod=true,cluster=teamcity-17753131-1731567227-103-n6cpu4-geo,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2876127073 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17753131-1731567227-103-n6cpu4-geo-0001 teamcity-17753131-1731567227-103-n6cpu4-geo-0002 teamcity-17753131-1731567227-103-n6cpu4-geo-0003]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-103-n6cpu4-geo-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test jepsen/bank/majority-ring-start-kill-2 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-14t10_13_42z,roachprod=true,cluster=teamcity-17753131-1731567227-121-n6cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1226239427 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17753131-1731567227-121-n6cpu4-0001 teamcity-17753131-1731567227-121-n6cpu4-0002 teamcity-17753131-1731567227-121-n6cpu4-0003 teamcity-17753131-1731567227-121-n6cpu4-0004 teamcity-17753131-1731567227-121-n6cpu4-0005 teamcity-17753131-1731567227-121-n6cpu4-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-121-n6cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17753131-1731567227-121-n6cpu4-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 720d706874e2424b32c7c9347f6d52140f83beee:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17772439-1731653577-93-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-15t12_43_04z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script1174451287 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17772439-1731653577-93-n4cpu96-0001 teamcity-17772439-1731653577-93-n4cpu96-0002 teamcity-17772439-1731653577-93-n4cpu96-0003 teamcity-17772439-1731653577-93-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17772439-1731653577-93-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 720d706874e2424b32c7c9347f6d52140f83beee:

test decommissionBench/nodes=4/warehouses=1000/duration=1h0m0s failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-11-17t09_43_01z,roachprod=true,cluster=teamcity-17801117-1731826265-48-n5cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script590778924 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17801117-1731826265-48-n5cpu16-0001 teamcity-17801117-1731826265-48-n5cpu16-0002 teamcity-17801117-1731826265-48-n5cpu16-0003 teamcity-17801117-1731826265-48-n5cpu16-0004 teamcity-17801117-1731826265-48-n5cpu16-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17801117-1731826265-48-n5cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17801117-1731826265-48-n5cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17801117-1731826265-48-n5cpu16-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20241115'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.16-rc @ 720d706874e2424b32c7c9347f6d52140f83beee:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-17t13_19_54z,roachprod=true,cluster=teamcity-17801117-1731826265-104-n4cpu96,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script3108130106 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17801117-1731826265-104-n4cpu96-0001 teamcity-17801117-1731826265-104-n4cpu96-0002 teamcity-17801117-1731826265-104-n4cpu96-0003 teamcity-17801117-1731826265-104-n4cpu96-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc]

This test on roachdash | Improve this report!