roachtest: cluster_creation failed

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ f3cb39621e8cf74af6c89cdd5ee5c67fcc7fd097:

test clearrange/checks=true failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17503899-1730267180-48-n10cpu16,lifetime=12h0m0s,arch=arm64,created=2024-10-30t08_52_44z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script565315176 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17503899-1730267180-48-n10cpu16-0001 teamcity-17503899-1730267180-48-n10cpu16-0002 teamcity-17503899-1730267180-48-n10cpu16-0003 teamcity-17503899-1730267180-48-n10cpu16-0004 teamcity-17503899-1730267180-48-n10cpu16-0005 teamcity-17503899-1730267180-48-n10cpu16-0006 teamcity-17503899-1730267180-48-n10cpu16-0007 teamcity-17503899-1730267180-48-n10cpu16-0008 teamcity-17503899-1730267180-48-n10cpu16-0009 teamcity-17503899-1730267180-48-n10cpu16-0010]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43766

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ f3cb39621e8cf74af6c89cdd5ee5c67fcc7fd097:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17503899-1730267180-103-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-10-30t12_20_57z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script777711937 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17503899-1730267180-103-n4cpu96-0001 teamcity-17503899-1730267180-103-n4cpu96-0002 teamcity-17503899-1730267180-103-n4cpu96-0003 teamcity-17503899-1730267180-103-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17503899-1730267180-103-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17503899-1730267180-103-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ 36a02ceba385b3eafa1aebd4747bac23d7bf59d3:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17523508-1730353400-97-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-10-31t12_35_23z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script3561048215 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17523508-1730353400-97-n4cpu96-0001 teamcity-17523508-1730353400-97-n4cpu96-0002 teamcity-17523508-1730353400-97-n4cpu96-0003 teamcity-17523508-1730353400-97-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17523508-1730353400-97-n4cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17523508-1730353400-97-n4cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17523508-1730353400-97-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ af43a079580443e896d18fac6c613e3cc0ca2757:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-03t12_52_06z,roachprod=true,cluster=teamcity-17568604-1730616189-98-n4cpu96 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script831523685 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17568604-1730616189-98-n4cpu96-0001 teamcity-17568604-1730616189-98-n4cpu96-0002 teamcity-17568604-1730616189-98-n4cpu96-0003 teamcity-17568604-1730616189-98-n4cpu96-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ af43a079580443e896d18fac6c613e3cc0ca2757:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-03t13_16_18z,roachprod=true,cluster=teamcity-17568604-1730616189-101-n5cpu96,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script721805131 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17568604-1730616189-101-n5cpu96-0001 teamcity-17568604-1730616189-101-n5cpu96-0002 teamcity-17568604-1730616189-101-n5cpu96-0003 teamcity-17568604-1730616189-101-n5cpu96-0004 teamcity-17568604-1730616189-101-n5cpu96-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17568604-1730616189-101-n5cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17568604-1730616189-101-n5cpu96-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17568604-1730616189-101-n5cpu96-0005].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'N2D_CPUS' exceeded.  Limit: 1000.0 in region us-east1.
    metric name = compute.googleapis.com/n2d_cpus
    limit name = N2D-CPUS-per-project-region
    limit = 1000.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ af43a079580443e896d18fac6c613e3cc0ca2757:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17578484-1730702658-35-n6cpu16,lifetime=12h0m0s,arch=arm64,created=2024-11-04t08_30_21z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3696980893 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17578484-1730702658-35-n6cpu16-0001 teamcity-17578484-1730702658-35-n6cpu16-0002 teamcity-17578484-1730702658-35-n6cpu16-0003 teamcity-17578484-1730702658-35-n6cpu16-0004 teamcity-17578484-1730702658-35-n6cpu16-0005 teamcity-17578484-1730702658-35-n6cpu16-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/control failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-12t09_10_55z,roachprod=true,cluster=teamcity-17711455-1731393922-33-n7cpu16,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script433804834 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17711455-1731393922-33-n7cpu16-0001 teamcity-17711455-1731393922-33-n7cpu16-0002 teamcity-17711455-1731393922-33-n7cpu16-0003 teamcity-17711455-1731393922-33-n7cpu16-0004 teamcity-17711455-1731393922-33-n7cpu16-0005 teamcity-17711455-1731393922-33-n7cpu16-0006 teamcity-17711455-1731393922-33-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17711455-1731393922-33-n7cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17711455-1731393922-33-n7cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17711455-1731393922-33-n7cpu16-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=processor/protocol=rangefeed/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-12t09_15_16z,roachprod=true,cluster=teamcity-17711455-1731393922-36-n7cpu16 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1955574050 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711455-1731393922-36-n7cpu16-0001 teamcity-17711455-1731393922-36-n7cpu16-0002 teamcity-17711455-1731393922-36-n7cpu16-0003 teamcity-17711455-1731393922-36-n7cpu16-0004 teamcity-17711455-1731393922-36-n7cpu16-0005 teamcity-17711455-1731393922-36-n7cpu16-0006 teamcity-17711455-1731393922-36-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-36-n7cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-36-n7cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-36-n7cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-36-n7cpu16-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test connection_latency/nodes=9/multiregion/password failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17711455-1731393922-41-n12cpu4-geo,lifetime=12h0m0s,arch=amd64,created=2024-11-12t09_30_48z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3035180453 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711455-1731393922-41-n12cpu4-geo-0001 teamcity-17711455-1731393922-41-n12cpu4-geo-0002 teamcity-17711455-1731393922-41-n12cpu4-geo-0003 teamcity-17711455-1731393922-41-n12cpu4-geo-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-41-n12cpu4-geo-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-41-n12cpu4-geo-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-41-n12cpu4-geo-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test disk-stalled/cgroup/read-write/logs-too=false failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-11-12t09_35_37z,roachprod=true,cluster=teamcity-17711455-1731393922-49-n4cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2004-focal-arm64-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script2794051671 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17711455-1731393922-49-n4cpu4-0001 teamcity-17711455-1731393922-49-n4cpu4-0002 teamcity-17711455-1731393922-49-n4cpu4-0003 teamcity-17711455-1731393922-49-n4cpu4-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test cdc/workload/kv100/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17711455-1731393922-51-n7cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-12t09_36_19z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2746529272 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17711455-1731393922-51-n7cpu16-0001 teamcity-17711455-1731393922-51-n7cpu16-0002 teamcity-17711455-1731393922-51-n7cpu16-0003 teamcity-17711455-1731393922-51-n7cpu16-0004 teamcity-17711455-1731393922-51-n7cpu16-0005 teamcity-17711455-1731393922-51-n7cpu16-0006 teamcity-17711455-1731393922-51-n7cpu16-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test drain-and-decommission/nodes=9 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-11-12t09_36_42z,roachprod=true,cluster=teamcity-17711455-1731393922-53-n9cpu4,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script1024174303 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17711455-1731393922-53-n9cpu4-0001 teamcity-17711455-1731393922-53-n9cpu4-0002 teamcity-17711455-1731393922-53-n9cpu4-0003 teamcity-17711455-1731393922-53-n9cpu4-0004 teamcity-17711455-1731393922-53-n9cpu4-0005 teamcity-17711455-1731393922-53-n9cpu4-0006 teamcity-17711455-1731393922-53-n9cpu4-0007 teamcity-17711455-1731393922-53-n9cpu4-0008 teamcity-17711455-1731393922-53-n9cpu4-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ d1924349a6d6acbb1cd7fb96c8e191c7061c3d37:

test clearrange/checks=false failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-12t10_05_12z,roachprod=true,cluster=teamcity-17711455-1731393922-69-n10cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3978270476 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711455-1731393922-69-n10cpu16-0001 teamcity-17711455-1731393922-69-n10cpu16-0002 teamcity-17711455-1731393922-69-n10cpu16-0003 teamcity-17711455-1731393922-69-n10cpu16-0004 teamcity-17711455-1731393922-69-n10cpu16-0005 teamcity-17711455-1731393922-69-n10cpu16-0006 teamcity-17711455-1731393922-69-n10cpu16-0007 teamcity-17711455-1731393922-69-n10cpu16-0008 teamcity-17711455-1731393922-69-n10cpu16-0009 teamcity-17711455-1731393922-69-n10cpu16-0010]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0008].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711455-1731393922-69-n10cpu16-0010].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17734171-1731480270-33-n7cpu16,lifetime=12h0m0s,arch=arm64,created=2024-11-13t08_29_19z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script112748724 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17734171-1731480270-33-n7cpu16-0001 teamcity-17734171-1731480270-33-n7cpu16-0002 teamcity-17734171-1731480270-33-n7cpu16-0003 teamcity-17734171-1731480270-33-n7cpu16-0004 teamcity-17734171-1731480270-33-n7cpu16-0005 teamcity-17734171-1731480270-33-n7cpu16-0006 teamcity-17734171-1731480270-33-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17734171-1731480270-33-n7cpu16-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.cluster_creation failed with artifacts on release-23.2.15-rc @ 0b7ccb1685a578536183187c4d664f3539601f60:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:753).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17734171-1731480270-95-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-13t12_18_19z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script3565839227 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17734171-1731480270-95-n4cpu96-0001 teamcity-17734171-1731480270-95-n4cpu96-0002 teamcity-17734171-1731480270-95-n4cpu96-0003 teamcity-17734171-1731480270-95-n4cpu96-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #133780 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.5-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #133805