search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.22k stars 3.82k forks source link

roachtest: cluster_creation failed #110990

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:745).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11846147-1695188585-156-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-09-20t18_40_57z --metadata-from-file startup-script=/tmp/gce-startup-script808177416 --project cockroach-ephemeral --boot-disk-size=32GB --zone eu-west-1b teamcity-11846147-1695188585-156-n9cpu8-geo-0007 teamcity-11846147-1695188585-156-n9cpu8-geo-0008 teamcity-11846147-1695188585-156-n9cpu8-geo-0009]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/eu-west-1b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-31719

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test kv0/enc=false/nodes=1/cpu=32 was skipped due to (test_runner.go:745).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-SS7VcBpJj3vvTzYlmbec_Qa_iBo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T05:59:50Z},{Key=Name,Value=teamcity-11864958-1695274927-16-n2cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-16-n2cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T05:59:50Z},{Key=Name,Value=teamcity-11864958-1695274927-16-n2cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-16-n2cpu32}] --user-data file:///tmp/aws-startup-script3096740984 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping750595630 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:745).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-SS7VcBpJj3vvTzYlmbec_Qa_iBo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:42:01Z},{Key=Name,Value=teamcity-11864958-1695274927-38-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-38-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:42:01Z},{Key=Name,Value=teamcity-11864958-1695274927-38-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-38-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4024286709 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4095179911 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test ycsb/F/nodes=3/cpu=32 was skipped due to (test_runner.go:745).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-SS7VcBpJj3vvTzYlmbec_Qa_iBo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:43:58Z},{Key=Name,Value=teamcity-11864958-1695274927-41-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-41-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:43:58Z},{Key=Name,Value=teamcity-11864958-1695274927-41-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script3876229202 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping551043476 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test ycsb/F/nodes=3/cpu=32/mvcc-range-keys=global was skipped due to (test_runner.go:745).runWorker: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0a14db46282743a66 --key-name teamcity-runner-SS7VcBpJj3vvTzYlmbec_Qa_iBo= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-09065271eb9d0144d --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11864958-1695274927-44-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:45:53Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-11864958-1695274927-44-n4cpu32-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-11864958-1695274927-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2023-09-21T07:45:53Z}] --user-data file:///tmp/aws-startup-script4174782450 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping403212978 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2c). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2b.: exit status 254

Parameters: ROACHTEST_cloud=aws , ROACHTEST_cpu=32 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test kv0/enc=false/nodes=3/cpu=96 was skipped due to (test_runner.go:745).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2-highcpu-96 --labels usage=roachtest,roachprod=true,cluster=teamcity-11864957-1695274981-136-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2023-09-21t18_12_47z --metadata-from-file startup-script=/tmp/gce-startup-script2924915117 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-11864957-1695274981-136-n4cpu96-0001 teamcity-11864957-1695274981-136-n4cpu96-0002 teamcity-11864957-1695274981-136-n4cpu96-0003 teamcity-11864957-1695274981-136-n4cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11864957-1695274981-136-n4cpu96-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11864957-1695274981-136-n4cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-11864957-1695274981-136-n4cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230817' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20230918'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 400000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 400000.0
    dimensions = region: us-east1
vm_family: N2
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=96 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cluster_creation failed with artifacts on release-23.1.11-rc.FROZEN @ cd8ef91e69525d5e1f4020e0cd57368778939772:

test restore/tpce/400GB/aws/nodes=9/cpus=8/zones=us-east-2b,us-west-2b,eu-west-1b was skipped due to (test_runner.go:745).runWorker: in provider: gce: Command: gcloud [compute instances create --subnet default --scopes cloud-platform --image ubuntu-2004-focal-v20230817 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=1000GB,auto-delete=yes --machine-type n2-standard-8 --labels usage=roachtest,roachprod=true,cluster=teamcity-11864957-1695274981-162-n9cpu8-geo,lifetime=12h0m0s,arch=amd64,created=2023-09-21t18_40_10z --metadata-from-file startup-script=/tmp/gce-startup-script1604777812 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east-2b teamcity-11864957-1695274981-162-n9cpu8-geo-0001 teamcity-11864957-1695274981-162-n9cpu8-geo-0002 teamcity-11864957-1695274981-162-n9cpu8-geo-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Permission denied on 'locations/us-east-2b' (or it may not exist).: exit status 1

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=8 , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #109982 roachtest: cluster_creation failed [FROZEN.branch-release-23.1.11-rc O-roachtest O-robot T-testeng X-infra-flake] - #108859 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-22.2] - #108653 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.9-rc.FROZEN] - #108629 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

renatolabs commented 1 year ago

Branch doesn't exist anymore, closing.