roachtest: cluster_creation failed

cockroach-teamcity commented 8 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:823).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

cloud=aws
coverageBuild=false
cpu=96
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44949

cockroach-teamcity commented 8 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test backup/2TB/n10cpu4 failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,arch=amd64,created=2024-11-27t07_13_37z,roachprod=true,cluster=teamcity-17935206-1732690277-20-n10cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script395700734 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935206-1732690277-20-n10cpu4-0001 teamcity-17935206-1732690277-20-n10cpu4-0002 teamcity-17935206-1732690277-20-n10cpu4-0003 teamcity-17935206-1732690277-20-n10cpu4-0004 teamcity-17935206-1732690277-20-n10cpu4-0005 teamcity-17935206-1732690277-20-n10cpu4-0006 teamcity-17935206-1732690277-20-n10cpu4-0007 teamcity-17935206-1732690277-20-n10cpu4-0008 teamcity-17935206-1732690277-20-n10cpu4-0009 teamcity-17935206-1732690277-20-n10cpu4-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17935206-1732690277-20-n10cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test cdc/initial-scan-only failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-27t07_55_36z,roachprod=true,cluster=teamcity-17935206-1732690277-34-n4cpu16 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2167107487 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935206-1732690277-34-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test cdc/kafka-chaos failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,created=2024-11-27t08_03_01z,roachprod=true,cluster=teamcity-17935206-1732690277-40-n4cpu16,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4013530610 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935206-1732690277-40-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test connection_latency/nodes=9/multiregion/certs failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17935206-1732690277-64-n12cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_01_47z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2653119966 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935206-1732690277-64-n12cpu4-geo-0001 teamcity-17935206-1732690277-64-n12cpu4-geo-0002 teamcity-17935206-1732690277-64-n12cpu4-geo-0003 teamcity-17935206-1732690277-64-n12cpu4-geo-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test connection_latency/nodes=9/multiregion/password failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,created=2024-11-27t09_07_52z,roachprod=true,cluster=teamcity-17935206-1732690277-68-n12cpu4-geo,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script359442 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935206-1732690277-68-n12cpu4-geo-0001 teamcity-17935206-1732690277-68-n12cpu4-geo-0002 teamcity-17935206-1732690277-68-n12cpu4-geo-0003 teamcity-17935206-1732690277-68-n12cpu4-geo-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=4
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test copy/bank/rows=1000000,nodes=9,txn=false failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,created=2024-11-27t09_09_08z,roachprod=true,cluster=teamcity-17935206-1732690277-71-n9cpu4,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2851498533 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935206-1732690277-71-n9cpu4-0001 teamcity-17935206-1732690277-71-n9cpu4-0002 teamcity-17935206-1732690277-71-n9cpu4-0003 teamcity-17935206-1732690277-71-n9cpu4-0004 teamcity-17935206-1732690277-71-n9cpu4-0005 teamcity-17935206-1732690277-71-n9cpu4-0006 teamcity-17935206-1732690277-71-n9cpu4-0007 teamcity-17935206-1732690277-71-n9cpu4-0008 teamcity-17935206-1732690277-71-n9cpu4-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935206-1732690277-71-n9cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test decommissionBench/nodes=4/warehouses=1000/while-down failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17935206-1732690277-77-n5cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_15_15z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2525719601 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17935206-1732690277-77-n5cpu16-0001 teamcity-17935206-1732690277-77-n5cpu16-0002 teamcity-17935206-1732690277-77-n5cpu16-0003 teamcity-17935206-1732690277-77-n5cpu16-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-77-n5cpu16-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=16
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test clearrange/zfs/checks=true failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-27t09_19_27z,roachprod=true,cluster=teamcity-17935206-1732690277-81-n10cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4180375976 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17935206-1732690277-81-n10cpu16-0001 teamcity-17935206-1732690277-81-n10cpu16-0002 teamcity-17935206-1732690277-81-n10cpu16-0003 teamcity-17935206-1732690277-81-n10cpu16-0004 teamcity-17935206-1732690277-81-n10cpu16-0005 teamcity-17935206-1732690277-81-n10cpu16-0006 teamcity-17935206-1732690277-81-n10cpu16-0007 teamcity-17935206-1732690277-81-n10cpu16-0008 teamcity-17935206-1732690277-81-n10cpu16-0009 teamcity-17935206-1732690277-81-n10cpu16-0010]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=16
runtimeAssertionsBuild=true
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test drain-and-decommission/nodes=9 failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,arch=amd64,created=2024-11-27t09_39_01z,roachprod=true,cluster=teamcity-17935206-1732690277-95-n9cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1987071409 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17935206-1732690277-95-n9cpu4-0001 teamcity-17935206-1732690277-95-n9cpu4-0002 teamcity-17935206-1732690277-95-n9cpu4-0003 teamcity-17935206-1732690277-95-n9cpu4-0004 teamcity-17935206-1732690277-95-n9cpu4-0005 teamcity-17935206-1732690277-95-n9cpu4-0006 teamcity-17935206-1732690277-95-n9cpu4-0007 teamcity-17935206-1732690277-95-n9cpu4-0008 teamcity-17935206-1732690277-95-n9cpu4-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-95-n9cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-95-n9cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-95-n9cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-95-n9cpu4-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=4
runtimeAssertionsBuild=true
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test failover/non-system/blackhole-recv/lease=expiration failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_45_30z,roachprod=true,cluster=teamcity-17935206-1732690277-101-n7cpu2 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script4112176258 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935206-1732690277-101-n7cpu2-0001 teamcity-17935206-1732690277-101-n7cpu2-0002 teamcity-17935206-1732690277-101-n7cpu2-0003 teamcity-17935206-1732690277-101-n7cpu2-0004 teamcity-17935206-1732690277-101-n7cpu2-0005 teamcity-17935206-1732690277-101-n7cpu2-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=2
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test failover/non-system/blackhole/lease=expiration failed: (test_runner.go:823).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17935206-1732690277-105-n7cpu2,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_49_53z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1133671779 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17935206-1732690277-105-n7cpu2-0001 teamcity-17935206-1732690277-105-n7cpu2-0002 teamcity-17935206-1732690277-105-n7cpu2-0003 teamcity-17935206-1732690277-105-n7cpu2-0004 teamcity-17935206-1732690277-105-n7cpu2-0005 teamcity-17935206-1732690277-105-n7cpu2-0006]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-105-n7cpu2-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-105-n7cpu2-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17935206-1732690277-105-n7cpu2-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=2
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 hours ago

roachtest.cluster_creation failed with artifacts on release-24.2.6-rc @ 9dab6cb9f71f21ef1e9c98640193035783ba9a4e:

test sysbench/oltp_insert/nodes=3/cpu=32/conc=256 failed: (test_runner.go:823).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

cloud=gce
coverageBuild=false
cpu=32
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136276 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.17-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #136275