search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

roachtest: cluster_creation failed #136276

Open cockroach-teamcity opened 3 days ago

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test alterpk-tpcc-500 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-27t07_48_42z,roachprod=true,cluster=teamcity-17935150-1732690698-21-n4cpu16 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1599979307 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935150-1732690698-21-n4cpu16-0001 teamcity-17935150-1732690698-21-n4cpu16-0002 teamcity-17935150-1732690698-21-n4cpu16-0003 teamcity-17935150-1732690698-21-n4cpu16-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-21-n4cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-21-n4cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-21-n4cpu16-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-44950

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test c2c/disconnect failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17935150-1732690698-27-n7cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-27t07_53_58z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1836418130 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935150-1732690698-27-n7cpu4-0001 teamcity-17935150-1732690698-27-n7cpu4-0002 teamcity-17935150-1732690698-27-n7cpu4-0003 teamcity-17935150-1732690698-27-n7cpu4-0004 teamcity-17935150-1732690698-27-n7cpu4-0005 teamcity-17935150-1732690698-27-n7cpu4-0006 teamcity-17935150-1732690698-27-n7cpu4-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test c2c/shutdown/dest/worker failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17935150-1732690698-30-n9cpu8,lifetime=12h0m0s,arch=amd64,created=2024-11-27t07_58_11z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script869440942 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935150-1732690698-30-n9cpu8-0001 teamcity-17935150-1732690698-30-n9cpu8-0002 teamcity-17935150-1732690698-30-n9cpu8-0003 teamcity-17935150-1732690698-30-n9cpu8-0004 teamcity-17935150-1732690698-30-n9cpu8-0005 teamcity-17935150-1732690698-30-n9cpu8-0006 teamcity-17935150-1732690698-30-n9cpu8-0007 teamcity-17935150-1732690698-30-n9cpu8-0008 teamcity-17935150-1732690698-30-n9cpu8-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-30-n9cpu8-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test c2c/shutdown/src/coordinator failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-27t08_03_00z,roachprod=true,cluster=teamcity-17935150-1732690698-33-n9cpu8 --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1403076200 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935150-1732690698-33-n9cpu8-0001 teamcity-17935150-1732690698-33-n9cpu8-0002 teamcity-17935150-1732690698-33-n9cpu8-0003 teamcity-17935150-1732690698-33-n9cpu8-0004 teamcity-17935150-1732690698-33-n9cpu8-0005 teamcity-17935150-1732690698-33-n9cpu8-0006 teamcity-17935150-1732690698-33-n9cpu8-0007 teamcity-17935150-1732690698-33-n9cpu8-0008 teamcity-17935150-1732690698-33-n9cpu8-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-33-n9cpu8-0006].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test cdc/scan/initial/nodes=5/cpu=16/rows=1G/ranges=100k/protocol=mux/format=json/sink=null failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=amd64,created=2024-11-27t09_02_52z,roachprod=true,cluster=teamcity-17935150-1732690698-46-n6cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-standard-16 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script965148966 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935150-1732690698-46-n6cpu16-0001 teamcity-17935150-1732690698-46-n6cpu16-0002 teamcity-17935150-1732690698-46-n6cpu16-0003 teamcity-17935150-1732690698-46-n6cpu16-0004 teamcity-17935150-1732690698-46-n6cpu16-0005 teamcity-17935150-1732690698-46-n6cpu16-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=arm64,created=2024-11-27t09_15_44z,roachprod=true,cluster=teamcity-17935150-1732690698-58-n7cpu16 --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script2201027749 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17935150-1732690698-58-n7cpu16-0001 teamcity-17935150-1732690698-58-n7cpu16-0002 teamcity-17935150-1732690698-58-n7cpu16-0003 teamcity-17935150-1732690698-58-n7cpu16-0004 teamcity-17935150-1732690698-58-n7cpu16-0005 teamcity-17935150-1732690698-58-n7cpu16-0006 teamcity-17935150-1732690698-58-n7cpu16-0007]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17935150-1732690698-58-n7cpu16-0007].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test follower-reads/mixed-version/single-region failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17935150-1732690698-68-n4cpu2,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_23_15z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3955632503 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935150-1732690698-68-n4cpu2-0001 teamcity-17935150-1732690698-68-n4cpu2-0002 teamcity-17935150-1732690698-68-n4cpu2-0003 teamcity-17935150-1732690698-68-n4cpu2-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test copy/bank/rows=1000000,nodes=9,txn=false failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17935150-1732690698-67-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_23_04z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script517587132 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935150-1732690698-67-n9cpu4-0001 teamcity-17935150-1732690698-67-n9cpu4-0002 teamcity-17935150-1732690698-67-n9cpu4-0003 teamcity-17935150-1732690698-67-n9cpu4-0004 teamcity-17935150-1732690698-67-n9cpu4-0005 teamcity-17935150-1732690698-67-n9cpu4-0006 teamcity-17935150-1732690698-67-n9cpu4-0007 teamcity-17935150-1732690698-67-n9cpu4-0008 teamcity-17935150-1732690698-67-n9cpu4-0009]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-67-n9cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-67-n9cpu4-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-67-n9cpu4-0006].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-67-n9cpu4-0007].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17935150-1732690698-67-n9cpu4-0009].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20230727' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20241119'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test drain-and-decommission/nodes=9 failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17935150-1732690698-86-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-27t09_49_46z --scopes cloud-platform --image ubuntu-2204-jammy-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2860917208 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17935150-1732690698-86-n9cpu4-0001 teamcity-17935150-1732690698-86-n9cpu4-0002 teamcity-17935150-1732690698-86-n9cpu4-0003 teamcity-17935150-1732690698-86-n9cpu4-0004 teamcity-17935150-1732690698-86-n9cpu4-0005 teamcity-17935150-1732690698-86-n9cpu4-0006 teamcity-17935150-1732690698-86-n9cpu4-0007 teamcity-17935150-1732690698-86-n9cpu4-0008 teamcity-17935150-1732690698-86-n9cpu4-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test disk-stalled/cgroup/write-only/logs-too=true failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-27t09_49_44z,roachprod=true,cluster=teamcity-17935150-1732690698-85-n4cpu4,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script1587754441 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17935150-1732690698-85-n4cpu4-0001 teamcity-17935150-1732690698-85-n4cpu4-0002 teamcity-17935150-1732690698-85-n4cpu4-0003 teamcity-17935150-1732690698-85-n4cpu4-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17935150-1732690698-85-n4cpu4-0002].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20241115'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

message: The zone 'projects/cockroach-ephemeral/zones/us-central1-a' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A t2a-standard-16 VM instance is currently unavailable in the us-central1-a
      zone. Consider trying your request in the us-central1-f, us-central1-b zone(s),
      which currently has capacity to accommodate your request. Alternatively, you
      can try your request again with a different VM hardware configuration or at
      a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: ''
      vmType: t2a-standard-16
      zone: us-central1-a
      zonesAvailable: us-central1-f,us-central1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-a' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A t2a-standard-16 VM instance is currently unavailable in the us-central1-a
      zone. Consider trying your request in the us-central1-f, us-central1-b zone(s),
      which currently has capacity to accommodate your request. Alternatively, you
      can try your request again with a different VM hardware configuration or at
      a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: ''
      vmType: t2a-standard-16
      zone: us-central1-a
      zonesAvailable: us-central1-f,us-central1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-central1-a' does not have
  enough resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 hours ago

roachtest.cluster_creation failed with artifacts on release-23.2.17-rc @ 1a524cd3ced15426926fb4898162ab43c7484d92:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/server=scheduler/protocol=mux/format=json/sink=null failed: (test_runner.go:752).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=arm64,created=2024-11-30t08_34_14z,roachprod=true,cluster=teamcity-17973204-1732949646-35-n7cpu16 --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20230727 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script1449714166 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17973204-1732949646-35-n7cpu16-0001 teamcity-17973204-1732949646-35-n7cpu16-0002 teamcity-17973204-1732949646-35-n7cpu16-0003 teamcity-17973204-1732949646-35-n7cpu16-0004 teamcity-17973204-1732949646-35-n7cpu16-0005 teamcity-17973204-1732949646-35-n7cpu16-0006 teamcity-17973204-1732949646-35-n7cpu16-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #136275 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.6-rc] - #136204 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.30-rc] - #136203 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.8-rc] - #136191 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.1-rc] - #135836 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #135144 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake] - #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake]

This test on roachdash | Improve this report!