search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.99k stars 3.79k forks source link

roachtest: cluster_creation failed #78601

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test tpccbench/nodes=3/cpu=16 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:24Z},{Key=Name,Value=teamcity-4698278-1648444789-10-n4cpu16-0002}] --user-data file:///tmp/aws-startup-script3706005379 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4036371810 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-10-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:48:24Z},{Key=Name,Value=teamcity-4698278-1648444789-10-n4cpu16-0002}] --user-data file:///tmp/aws-startup-script3706005379 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4036371810 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1]

/cc @cockroachdb/dev-inf

This test on roachdash | Improve this report!

Jira issue: CRDB-14193

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test tpccbench/nodes=3/cpu=16/no-admission was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4698278-1648444789-13-n4cpu16-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-13-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:49:27Z}] --user-data file:///tmp/aws-startup-script3564196625 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3289024918 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c5d.4xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4698278-1648444789-13-n4cpu16-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-13-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T12:49:27Z}] --user-data file:///tmp/aws-startup-script3564196625 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3289024918 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c5d.4xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get c5d.4xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T15:20:57Z},{Key=Name,Value=teamcity-4698278-1648444789-17-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-17-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4078450040 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3290289358 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-03-28T15:20:57Z},{Key=Name,Value=teamcity-4698278-1648444789-17-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-17-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4078450040 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3290289358 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 32b45c4bcf1ab41f0ba3abd36cb670eea7f450fd:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T15:21:54Z},{Key=Name,Value=teamcity-4698278-1648444789-20-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script1956457138 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping389602635 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-K9sxDtvNHA4eytNlf-coAmkqWJs= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4698278-1648444789-20-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-03-28T15:21:54Z},{Key=Name,Value=teamcity-4698278-1648444789-20-n4cpu96-0002}] --user-data file:///tmp/aws-startup-script1956457138 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping389602635 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 0e6878b6140cb5cc7c799fdf576b3d565a54fd2e:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:15:37Z},{Key=Name,Value=teamcity-4766499-1648876713-12-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3560179776 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping360008570 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-12-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:15:37Z},{Key=Name,Value=teamcity-4766499-1648876713-12-n4cpu96-0001}] --user-data file:///tmp/aws-startup-script3560179776 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping360008570 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 0e6878b6140cb5cc7c799fdf576b3d565a54fd2e:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4766499-1648876713-15-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-15-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:16:35Z}] --user-data file:///tmp/aws-startup-script309430335 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1555857328 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-iVwxzIh71NgwXPrw6hLVgSGQqB0= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Name,Value=teamcity-4766499-1648876713-15-n4cpu96-0002},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4766499-1648876713-15-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-02T18:16:35Z}] --user-data file:///tmp/aws-startup-script309430335 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1555857328 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 72b71b6920fe5b6dde2c7229b4edc2f95432839e:

The test failed on branch=master, cloud=aws:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-06T18:03:52Z},{Key=Name,Value=teamcity-4807858-1649222398-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script3252835754 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2609775494 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-11-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=Created,Value=2022-04-06T18:03:52Z},{Key=Name,Value=teamcity-4807858-1649222398-11-n4cpu96-0003}] --user-data file:///tmp/aws-startup-script3252835754 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2609775494 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 72b71b6920fe5b6dde2c7229b4edc2f95432839e:

The test failed on branch=master, cloud=aws:
test kv95/enc=false/nodes=3/cpu=96 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-04-06T18:04:49Z},{Key=Name,Value=teamcity-4807858-1649222398-14-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-14-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script180643413 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1999646734 --output json: stderr: : exit status 254
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: aws
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:194
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runJSONCommand
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/support.go:204
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).runInstance
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:969
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws.(*Provider).Create.func1
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/aws/aws.go:443
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m5d.24xlarge --image-id ami-02fc6052104add5ae --key-name teamcity-runner-vH_ZEImoEFvuuLwGxire-qhkHPc= --region us-east-2 --security-group-ids sg-0319fc9c9599a6145 --subnet-id subnet-0775beaa2f4c74f1c --tag-specifications ResourceType=instance,Tags=[{Key=Created,Value=2022-04-06T18:04:49Z},{Key=Name,Value=teamcity-4807858-1649222398-14-n4cpu96-0001},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-4807858-1649222398-14-n4cpu96},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script180643413 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1999646734 --output json: stderr: 
          | An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m5d.24xlarge capacity in the Availability Zone you requested (us-east-2b). Our system will be working on provisioning additional capacity. You can currently get m5d.24xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2a, us-east-2c.
        Wraps: (5) exit status 254
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 7a97bdd02c62787f40d79d5a2cd3439d8c9c8a1b:

The test failed on branch=master, cloud=gce:
test kv0/enc=false/nodes=3/cpu=96 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831888-1649395225-70-n4cpu96,lifetime=12h0m0s,created=2022-04-08t10_51_00z --metadata-from-file startup-script=/tmp/gce-startup-script3623791391 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels roachprod=true,cluster=teamcity-4831888-1649395225-70-n4cpu96,lifetime=12h0m0s,created=2022-04-08t10_51_00z --metadata-from-file startup-script=/tmp/gce-startup-script3623791391 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-70-n4cpu96-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-70-n4cpu96-0002].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ 7a97bdd02c62787f40d79d5a2cd3439d8c9c8a1b:

The test failed on branch=master, cloud=gce:
test kv50/enc=false/nodes=4/cpu=96/batch=64 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4831888-1649395225-100-n5cpu96,lifetime=12h0m0s,created=2022-04-08t15_01_29z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script595119232 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-96 --labels cluster=teamcity-4831888-1649395225-100-n5cpu96,lifetime=12h0m0s,created=2022-04-08t15_01_29z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script595119232 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4831888-1649395225-100-n5cpu96-0001].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - The zone 'projects/cockroach-ephemeral/zones/us-east1-b' does not have enough resources available to fulfill the request.  '(resource type:compute)'.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test alterpk-tpcc-500 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-23-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_05z --metadata-from-file startup-script=/tmp/gce-startup-script1162684711 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-23-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_05z --metadata-from-file startup-script=/tmp/gce-startup-script1162684711 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test asyncpg was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-25-n1cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_22z --metadata-from-file startup-script=/tmp/gce-startup-script1289394172 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-25-n1cpu16,lifetime=12h0m0s,created=2022-04-15t05_51_22z --metadata-from-file startup-script=/tmp/gce-startup-script1289394172 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test autoupgrade was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_51_52z,roachprod=true,cluster=teamcity-4897747-1649999913-31-n5cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1727293541 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t05_51_52z,roachprod=true,cluster=teamcity-4897747-1649999913-31-n5cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1727293541 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backup/2TB/n10cpu4 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-37-n10cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_24z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2921408233 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-37-n10cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_24z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2921408233 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backup/KMS/AWS/n3cpu4 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-38-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_25z --metadata-from-file startup-script=/tmp/gce-startup-script337372255 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-38-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_25z --metadata-from-file startup-script=/tmp/gce-startup-script337372255 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test backupTPCC was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-40-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_37z --metadata-from-file startup-script=/tmp/gce-startup-script2392853458 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-40-n3cpu4,lifetime=12h0m0s,created=2022-04-15t05_52_37z --metadata-from-file startup-script=/tmp/gce-startup-script2392853458 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/crdb-chaos was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-48-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_53_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1777832219 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-48-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_53_48z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1777832219 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/initial-scan was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-53-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_54_57z --metadata-from-file startup-script=/tmp/gce-startup-script2645437982 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-53-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_54_57z --metadata-from-file startup-script=/tmp/gce-startup-script2645437982 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/cloud-sink-gcs/rangefeed=true was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t05_53_36z,roachprod=true,cluster=teamcity-4897747-1649999913-47-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script87802300 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels created=2022-04-15t05_53_36z,roachprod=true,cluster=teamcity-4897747-1649999913-47-n4cpu16,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script87802300 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-47-n4cpu16-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/ledger was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-60-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_58_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3173295066 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-60-n4cpu16,lifetime=12h0m0s,created=2022-04-15t05_58_26z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3173295066 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-60-n4cpu16-0001].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/tpcc-1000 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-64-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_01_51z --metadata-from-file startup-script=/tmp/gce-startup-script496068105 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-64-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_01_51z --metadata-from-file startup-script=/tmp/gce-startup-script496068105 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test cdc/sink-chaos was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-61-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_00_57z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script920124956 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels cluster=teamcity-4897747-1649999913-61-n4cpu16,lifetime=12h0m0s,created=2022-04-15t06_00_57z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script920124956 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-61-n4cpu16-0002].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test clearrange/checks=true was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-69-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_04_26z --metadata-from-file startup-script=/tmp/gce-startup-script3341602624 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-69-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_04_26z --metadata-from-file startup-script=/tmp/gce-startup-script3341602624 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test clearrange/checks=false was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-71-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_05_24z --metadata-from-file startup-script=/tmp/gce-startup-script3984140150 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-71-n10cpu16,lifetime=12h0m0s,created=2022-04-15t06_05_24z --metadata-from-file startup-script=/tmp/gce-startup-script3984140150 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-71-n10cpu16-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-71-n10cpu16-0009].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-71-n10cpu16-0010].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test copy/bank/rows=1000000,nodes=9,txn=false was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-85-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_21_25z --metadata-from-file startup-script=/tmp/gce-startup-script3889817523 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-85-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_21_25z --metadata-from-file startup-script=/tmp/gce-startup-script3889817523 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test django was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-90-n1cpu16,lifetime=12h0m0s,created=2022-04-15t06_24_56z --metadata-from-file startup-script=/tmp/gce-startup-script929413433 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-highcpu-16 --labels roachprod=true,cluster=teamcity-4897747-1649999913-90-n1cpu16,lifetime=12h0m0s,created=2022-04-15t06_24_56z --metadata-from-file startup-script=/tmp/gce-startup-script929413433 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test drain-and-decommission/nodes=9 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-95-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_28_33z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1669138748 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-95-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_28_33z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1669138748 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test drop/tpcc/w=100,nodes=9 was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-97-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script600317298 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-97-n9cpu4,lifetime=12h0m0s,created=2022-04-15t06_31_13z --metadata-from-file startup-script=/tmp/gce-startup-script600317298 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-97-n9cpu4-0005].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test import/tpcc/warehouses=1000/nodes=32 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-106-n32cpu4,lifetime=12h0m0s,created=2022-04-15t07_29_19z --metadata-from-file startup-script=/tmp/gce-startup-script2742350951 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-106-n32cpu4,lifetime=12h0m0s,created=2022-04-15t07_29_19z --metadata-from-file startup-script=/tmp/gce-startup-script2742350951 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0002].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0005].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0009].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0010].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0013].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0014].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0015].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0016].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0019].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0021].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0022].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0023].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0024].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0026].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0027].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0028].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0030].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-106-n32cpu4-0031].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/bank/start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_19_52z,roachprod=true,cluster=teamcity-4897747-1649999913-114-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1819909231 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_19_52z,roachprod=true,cluster=teamcity-4897747-1649999913-114-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script1819909231 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-114-n6cpu4-0003].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/g2/split was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-117-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_25_46z --metadata-from-file startup-script=/tmp/gce-startup-script851278124 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-117-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_25_46z --metadata-from-file startup-script=/tmp/gce-startup-script851278124 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/g2/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_29_19z,roachprod=true,cluster=teamcity-4897747-1649999913-122-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2735540773 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_29_19z,roachprod=true,cluster=teamcity-4897747-1649999913-122-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script2735540773 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/g2/start-stop-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_30_20z,roachprod=true,cluster=teamcity-4897747-1649999913-125-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script953929258 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_30_20z,roachprod=true,cluster=teamcity-4897747-1649999913-125-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script953929258 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/monotonic/majority-ring was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-130-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_32_48z --metadata-from-file startup-script=/tmp/gce-startup-script2087071177 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-130-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_32_48z --metadata-from-file startup-script=/tmp/gce-startup-script2087071177 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/monotonic/parts-start-kill-2 was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-128-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_31_30z --metadata-from-file startup-script=/tmp/gce-startup-script298791299 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-128-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_31_30z --metadata-from-file startup-script=/tmp/gce-startup-script298791299 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-128-n6cpu4-0001].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-128-n6cpu4-0005].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/monotonic/start-kill-2 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_33_19z,roachprod=true,cluster=teamcity-4897747-1649999913-133-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script2821694925 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_33_19z,roachprod=true,cluster=teamcity-4897747-1649999913-133-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script2821694925 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-133-n6cpu4-0004].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/monotonic/start-stop-2 was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-135-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_33_57z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2332840536 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-135-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_33_57z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script2332840536 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-135-n6cpu4-0003].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-135-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/multi-register/majority-ring was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-142-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_36_49z --metadata-from-file startup-script=/tmp/gce-startup-script1713822056 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-142-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_36_49z --metadata-from-file startup-script=/tmp/gce-startup-script1713822056 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/monotonic/strobe-skews was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-138-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_03z --metadata-from-file startup-script=/tmp/gce-startup-script1904267245 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-138-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_35_03z --metadata-from-file startup-script=/tmp/gce-startup-script1904267245 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-138-n6cpu4-0005].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/multi-register/majority-ring-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_37_34z,roachprod=true,cluster=teamcity-4897747-1649999913-145-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script322860468 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels lifetime=12h0m0s,created=2022-04-15t08_37_34z,roachprod=true,cluster=teamcity-4897747-1649999913-145-n6cpu4 --metadata-from-file startup-script=/tmp/gce-startup-script322860468 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-145-n6cpu4-0003].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/majority-ring was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-153-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_41_06z --metadata-from-file startup-script=/tmp/gce-startup-script3672943307 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-153-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_41_06z --metadata-from-file startup-script=/tmp/gce-startup-script3672943307 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/multi-register/split was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_40_19z,roachprod=true,cluster=teamcity-4897747-1649999913-148-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script86159597 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels created=2022-04-15t08_40_19z,roachprod=true,cluster=teamcity-4897747-1649999913-148-n6cpu4,lifetime=12h0m0s --metadata-from-file startup-script=/tmp/gce-startup-script86159597 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-148-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/multi-register/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-152-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_40_56z --metadata-from-file startup-script=/tmp/gce-startup-script1652345756 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-152-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_40_56z --metadata-from-file startup-script=/tmp/gce-startup-script1652345756 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-152-n6cpu4-0005].
          | Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-152-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/parts-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-162-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_43_54z --metadata-from-file startup-script=/tmp/gce-startup-script3024863363 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-162-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_43_54z --metadata-from-file startup-script=/tmp/gce-startup-script3024863363 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/split was skipped due to   test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-164-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_13z --metadata-from-file startup-script=/tmp/gce-startup-script2397983023 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-164-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_13z --metadata-from-file startup-script=/tmp/gce-startup-script2397983023 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-166-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_26z --metadata-from-file startup-script=/tmp/gce-startup-script547810663 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-166-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_44_26z --metadata-from-file startup-script=/tmp/gce-startup-script547810663 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/majority-ring-start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-156-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_42_15z --metadata-from-file startup-script=/tmp/gce-startup-script2000015254 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-156-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_42_15z --metadata-from-file startup-script=/tmp/gce-startup-script2000015254 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-4897747-1649999913-156-n6cpu4-0006].
          | WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/sequential/majority-ring was skipped due to     test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-171-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_21z --metadata-from-file startup-script=/tmp/gce-startup-script3539297858 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-171-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_21z --metadata-from-file startup-script=/tmp/gce-startup-script3539297858 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/sequential/parts-start-kill-2 was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-174-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_47z --metadata-from-file startup-script=/tmp/gce-startup-script15066698 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels roachprod=true,cluster=teamcity-4897747-1649999913-174-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_45_47z --metadata-from-file startup-script=/tmp/gce-startup-script15066698 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/sequential/start-kill-2 was skipped due to  test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-179-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_46_58z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3775125981 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-179-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_46_58z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script3775125981 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

roachtest.cluster_creation failed with artifacts on master @ b99291249d249ede65e732f89d3b3df8bf5472a4:

The test failed on branch=master, cloud=gce:
test jepsen/register/strobe-skews was skipped due to    test_runner.go:585,test_runner.go:260,stopper.go:494: in provider: gce: Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-182-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_47_27z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1973072239 --project cockroach-ephemeral --boot-disk-size=32GB]: exit status 1
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm.ForProvider
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/vm.go:374
          | [...repeated from below...]
        Wraps: (2) in provider: gce
        Wraps: (3) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce.(*Provider).Create.func2
          |     github.com/cockroachdb/cockroach/pkg/roachprod/vm/gce/gcloud.go:524
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (4) Command: gcloud [compute instances create --subnet default --maintenance-policy MIGRATE --scopes default,storage-rw --image ubuntu-2004-focal-v20210603 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --local-ssd interface=NVME --machine-type n1-standard-4 --labels cluster=teamcity-4897747-1649999913-182-n6cpu4,lifetime=12h0m0s,created=2022-04-15t08_47_27z,roachprod=true --metadata-from-file startup-script=/tmp/gce-startup-script1973072239 --project cockroach-ephemeral --boot-disk-size=32GB]
          | Output: WARNING: Some requests generated warnings:
          |  - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
          |  - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20210603' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2004-focal-v20220404'.
          |
          | ERROR: (gcloud.compute.instances.create) Could not fetch resource:
          |  - Quota 'CPUS' exceeded.  Limit: 3600.0 in region us-east1.
        Wraps: (5) exit status 1
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *exec.ExitError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #79984 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot branch-release-22.1.0 release-blocker] - #78035 roachtest: cluster_creation failed [C-test-failure O-roachtest O-robot T-testeng branch-release-22.1]

This test on roachdash | Improve this report!