cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

roachtest: cluster_creation failed #78035

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 2d49a7bd0d1ebeff93277233695411d25b2a5d39:

The test failed on branch=release-22.1, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:32:28Z},{Key=Name,Value=teamcity-4594125-1647494291-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script413225093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping769912648 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:32:28Z},{Key=Name,Value=teamcity-4594125-1647494291-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script413225093 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping769912648 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-13932

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 2d49a7bd0d1ebeff93277233695411d25b2a5d39:

The test failed on branch=release-22.1, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-14-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:33:32Z},{Key=Name,Value=teamcity-4594125-1647494291-14-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script880572208 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2964459769 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-38mJKEnGsabTBGqDXCF5gmtmzp4= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4594125-1647494291-14-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-17T17:33:32Z},{Key=Name,Value=teamcity-4594125-1647494291-14-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script880572208 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2964459769 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

The test failed on branch=release-22.1, cloud=aws:
test tpccbench/nodes=3/cpu=16 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:19Z},{Key=Name,Value=teamcity-4698289-1648444778-10-n4cpu16-0004}] --user-data file:///tmp/aws-startup-script2540826881 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping17736695 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:19Z},{Key=Name,Value=teamcity-4698289-1648444778-10-n4cpu16-0004}] --user-data file:///tmp/aws-startup-script2540826881 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping17736695 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ b5e0ec1844e63c4dc43dc7a9b1605b37c270ecbe:

The test failed on branch=release-22.1, cloud=aws:
test tpccbench/nodes=3/cpu=16/no-admission was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T12:49:18Z},{Key=Name,Value=teamcity-4698289-1648444778-13-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-13-n4cpu16},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1542903493 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3406362205 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-YbgTlaV6QVwqR9Y6Y-cChhO7sRI= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T12:49:18Z},{Key=Name,Value=teamcity-4698289-1648444778-13-n4cpu16-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698289-1648444778-13-n4cpu16},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1542903493 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3406362205 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-master release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 87e23b5a6de65b0f184ad71e6706eebd65ec6303:

The test failed on branch=release-22.1, cloud=gce:
test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831899-1649395134-94-n5cpu96,lifetime=12h0m0s,created=2022-04-08t14_44_10z --metadata-from-file startup-script=/tmp/gce-startup-script1657207299 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831899-1649395134-94-n5cpu96,lifetime=12h0m0s,created=2022-04-08t14_44_10z --metadata-from-file startup-script=/tmp/gce-startup-script1657207299 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test backup/KMS/AWS/n3cpu4 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t05_50_55z,roachprod=true,cluster=teamcity-4897758-1649999866-25-n3cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script278218323 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t05_50_55z,roachprod=true,cluster=teamcity-4897758-1649999866-25-n3cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script278218323 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-25-n3cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-25-n3cpu4-0002].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test backupTPCC was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_54_27z,roachprod=true,cluster=teamcity-4897758-1649999866-30-n3cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script140068770 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_54_27z,roachprod=true,cluster=teamcity-4897758-1649999866-30-n3cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script140068770 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cancel/tpch/distsql/queries=7,9,20,21,nodes=3 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-35-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_57_53z --metadata-from-file startup-script=/tmp/gce-startup-script3136905862 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-35-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_57_53z --metadata-from-file startup-script=/tmp/gce-startup-script3136905862 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-35-n3cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-35-n3cpu4-0002].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/cloud-sink-gcs/rangefeed=true was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897758-1649999866-38-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_06_05z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script871450961 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897758-1649999866-38-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_06_05z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script871450961 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/crdb-chaos was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-43-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_11_54z --metadata-from-file startup-script=/tmp/gce-startup-script348927181 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-43-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_11_54z --metadata-from-file startup-script=/tmp/gce-startup-script348927181 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-43-n4cpu16-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-43-n4cpu16-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/initial-scan was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_13_04z,roachprod=true,cluster=teamcity-4897758-1649999866-44-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1396308951 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_13_04z,roachprod=true,cluster=teamcity-4897758-1649999866-44-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script1396308951 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-44-n4cpu16-0002].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/sink-chaos was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_23_56z,roachprod=true,cluster=teamcity-4897758-1649999866-50-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4178551782 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t06_23_56z,roachprod=true,cluster=teamcity-4897758-1649999866-50-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script4178551782 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-50-n4cpu16-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test clearrange/checks=true was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-60-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script2157506098 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-60-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script2157506098 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test clearrange/checks=false was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-58-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_30_05z --metadata-from-file startup-script=/tmp/gce-startup-script3674307539 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-58-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_30_05z --metadata-from-file startup-script=/tmp/gce-startup-script3674307539 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-58-n10cpu16-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test cdc/tpcc-1000 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-63-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_32_10z --metadata-from-file startup-script=/tmp/gce-startup-script514418152 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897758-1649999866-63-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_32_10z --metadata-from-file startup-script=/tmp/gce-startup-script514418152 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-63-n4cpu16-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test copy/bank/rows=1000000,nodes=9,txn=false was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-71-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_39_49z --metadata-from-file startup-script=/tmp/gce-startup-script3995681688 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-71-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_39_49z --metadata-from-file startup-script=/tmp/gce-startup-script3995681688 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test decommission/randomized was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-76-n6cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_03z --metadata-from-file startup-script=/tmp/gce-startup-script3998134501 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-76-n6cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_03z --metadata-from-file startup-script=/tmp/gce-startup-script3998134501 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test drain-and-decommission/nodes=9 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-77-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_11z --metadata-from-file startup-script=/tmp/gce-startup-script1738687397 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-77-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_41_11z --metadata-from-file startup-script=/tmp/gce-startup-script1738687397 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test follower-reads/mixed-version/single-region was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-2 --labels roachprod=true,cluster=teamcity-4897758-1649999866-82-n3cpu2,lifetime=12h0m0s,created=2022-04-15t06_44_51z --metadata-from-file startup-script=/tmp/gce-startup-script3153286598 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-2 --labels roachprod=true,cluster=teamcity-4897758-1649999866-82-n3cpu2,lifetime=12h0m0s,created=2022-04-15t06_44_51z --metadata-from-file startup-script=/tmp/gce-startup-script3153286598 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test drop/tpcc/w=100,nodes=9 was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-84-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_46_56z --metadata-from-file startup-script=/tmp/gce-startup-script3813097881 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-84-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_46_56z --metadata-from-file startup-script=/tmp/gce-startup-script3813097881 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0008].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-84-n9cpu4-0009].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test follower-reads/survival=region/locality=global/reads=bounded-staleness was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-89-n6cpu4-geo,lifetime=12h0m0s,created=2022-04-15t06_50_58z --metadata-from-file startup-script=/tmp/gce-startup-script134413619 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-89-n6cpu4-geo,lifetime=12h0m0s,created=2022-04-15t06_50_58z --metadata-from-file startup-script=/tmp/gce-startup-script134413619 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test import/tpcc/warehouses=1000/nodes=32 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-98-n32cpu4,lifetime=12h0m0s,created=2022-04-15t07_24_34z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script457544094 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-98-n32cpu4,lifetime=12h0m0s,created=2022-04-15t07_24_34z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script457544094 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/bank/majority-ring-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-106-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_23_29z --metadata-from-file startup-script=/tmp/gce-startup-script3747578630 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-106-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_23_29z --metadata-from-file startup-script=/tmp/gce-startup-script3747578630 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/bank/parts-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-109-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_24_41z --metadata-from-file startup-script=/tmp/gce-startup-script324140979 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-109-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_24_41z --metadata-from-file startup-script=/tmp/gce-startup-script324140979 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/bank/start-stop-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_28_09z,roachprod=true,cluster=teamcity-4897758-1649999866-114-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2135021570 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_28_09z,roachprod=true,cluster=teamcity-4897758-1649999866-114-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2135021570 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/bank/start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_28_12z,roachprod=true,cluster=teamcity-4897758-1649999866-115-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script2564341786 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_28_12z,roachprod=true,cluster=teamcity-4897758-1649999866-115-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script2564341786 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/majority-ring was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-120-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_29_20z --metadata-from-file startup-script=/tmp/gce-startup-script263926958 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-120-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_29_20z --metadata-from-file startup-script=/tmp/gce-startup-script263926958 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/parts-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-123-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_30_31z --metadata-from-file startup-script=/tmp/gce-startup-script4187807908 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-123-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_30_31z --metadata-from-file startup-script=/tmp/gce-startup-script4187807908 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/majority-ring-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-125-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_31_45z --metadata-from-file startup-script=/tmp/gce-startup-script617267899 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-125-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_31_45z --metadata-from-file startup-script=/tmp/gce-startup-script617267899 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/split was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-129-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_33_59z --metadata-from-file startup-script=/tmp/gce-startup-script2409443235 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-129-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_33_59z --metadata-from-file startup-script=/tmp/gce-startup-script2409443235 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-133-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_26z --metadata-from-file startup-script=/tmp/gce-startup-script1155220194 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-133-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_26z --metadata-from-file startup-script=/tmp/gce-startup-script1155220194 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/g2/strobe-skews was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-132-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_09z --metadata-from-file startup-script=/tmp/gce-startup-script4252919447 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-132-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_09z --metadata-from-file startup-script=/tmp/gce-startup-script4252919447 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-132-n6cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-132-n6cpu4-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-132-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/monotonic/majority-ring was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-138-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_38_57z --metadata-from-file startup-script=/tmp/gce-startup-script436137151 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-138-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_38_57z --metadata-from-file startup-script=/tmp/gce-startup-script436137151 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/monotonic/parts-start-kill-2 was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_41_06z,roachprod=true,cluster=teamcity-4897758-1649999866-141-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script3517780811 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_41_06z,roachprod=true,cluster=teamcity-4897758-1649999866-141-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script3517780811 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/monotonic/start-stop-2 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-149-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_35z --metadata-from-file startup-script=/tmp/gce-startup-script1287180321 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-149-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_35z --metadata-from-file startup-script=/tmp/gce-startup-script1287180321 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/monotonic/split was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-145-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_42_27z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1004853069 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-145-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_42_27z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1004853069 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-145-n6cpu4-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-145-n6cpu4-0003].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/majority-ring-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-155-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_42z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2709864943 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-155-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_42z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2709864943 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/majority-ring was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_47_34z,roachprod=true,cluster=teamcity-4897758-1649999866-160-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script3798284045 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_47_34z,roachprod=true,cluster=teamcity-4897758-1649999866-160-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script3798284045 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/parts-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-156-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_58z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2189224082 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-156-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_58z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2189224082 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-156-n6cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-156-n6cpu4-0004].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-156-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/start-stop-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-164-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_48_42z --metadata-from-file startup-script=/tmp/gce-startup-script1693369152 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-164-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_48_42z --metadata-from-file startup-script=/tmp/gce-startup-script1693369152 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_46_54z,roachprod=true,cluster=teamcity-4897758-1649999866-159-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1014436199 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_46_54z,roachprod=true,cluster=teamcity-4897758-1649999866-159-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1014436199 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-159-n6cpu4-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-159-n6cpu4-0004].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-159-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/register/parts-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-174-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_50_24z --metadata-from-file startup-script=/tmp/gce-startup-script968862489 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-174-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_50_24z --metadata-from-file startup-script=/tmp/gce-startup-script968862489 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/multi-register/strobe-skews was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-167-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_49_08z --metadata-from-file startup-script=/tmp/gce-startup-script639641008 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-167-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_49_08z --metadata-from-file startup-script=/tmp/gce-startup-script639641008 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-167-n6cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-167-n6cpu4-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-167-n6cpu4-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-167-n6cpu4-0004].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-167-n6cpu4-0005].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/register/majority-ring-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-177-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_52_19z --metadata-from-file startup-script=/tmp/gce-startup-script788826842 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-177-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_52_19z --metadata-from-file startup-script=/tmp/gce-startup-script788826842 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/register/strobe-skews was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-182-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_55_49z --metadata-from-file startup-script=/tmp/gce-startup-script2096856114 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-182-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_55_49z --metadata-from-file startup-script=/tmp/gce-startup-script2096856114 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/register/split was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-184-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_56_17z --metadata-from-file startup-script=/tmp/gce-startup-script628685325 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-184-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_56_17z --metadata-from-file startup-script=/tmp/gce-startup-script628685325 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/sequential/majority-ring-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-187-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_57_27z --metadata-from-file startup-script=/tmp/gce-startup-script919645111 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-187-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_57_27z --metadata-from-file startup-script=/tmp/gce-startup-script919645111 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-187-n6cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-187-n6cpu4-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/sequential/majority-ring was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-190-n6cpu4,lifetime=12h0m0s,created=2022-04-15t09_01_42z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3284528121 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897758-1649999866-190-n6cpu4,lifetime=12h0m0s,created=2022-04-15t09_01_42z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3284528121 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test jepsen/sequential/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-193-n6cpu4,lifetime=12h0m0s,created=2022-04-15t09_03_20z --metadata-from-file startup-script=/tmp/gce-startup-script1028157447 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897758-1649999866-193-n6cpu4,lifetime=12h0m0s,created=2022-04-15t09_03_20z --metadata-from-file startup-script=/tmp/gce-startup-script1028157447 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4897758-1649999866-207-n4cpu96,lifetime=12h0m0s,created=2022-04-15t09_19_45z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1558372187 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4897758-1649999866-207-n4cpu96,lifetime=12h0m0s,created=2022-04-15t09_19_45z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1558372187 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on release-22.1 @ 8c867d72534db2c64beb1a36bde9f8345bfd5e9f:

The test failed on branch=release-22.1, cloud=gce:
test kv0/enc=false/nodes=3/batch=16 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-4897758-1649999866-212-n4cpu8,lifetime=12h0m0s,created=2022-04-15t09_24_10z --metadata-from-file startup-script=/tmp/gce-startup-script374254529 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-8 --labels roachprod=true,cluster=teamcity-4897758-1649999866-212-n4cpu8,lifetime=12h0m0s,created=2022-04-15t09_24_10z --metadata-from-file startup-script=/tmp/gce-startup-script374254529 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-212-n4cpu8-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-212-n4cpu8-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897758-1649999866-212-n4cpu8-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78601 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-master]

This test on roachdash | Improve this report!