roachtest: cluster_creation failed

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43745

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test sysbench/oltp_write_only/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test cdc/workload/kv0/nodes=5/cpu=16/ranges=100/control failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17523568-1730353731-38-n7cpu16,lifetime=12h0m0s,arch=arm64,created=2024-10-31t06_58_06z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script3026596122 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-f teamcity-17523568-1730353731-38-n7cpu16-0001 teamcity-17523568-1730353731-38-n7cpu16-0002 teamcity-17523568-1730353731-38-n7cpu16-0003 teamcity-17523568-1730353731-38-n7cpu16-0004 teamcity-17523568-1730353731-38-n7cpu16-0005 teamcity-17523568-1730353731-38-n7cpu16-0006 teamcity-17523568-1730353731-38-n7cpu16-0007]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv95/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test ycsb/C/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:816).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-qFqjpTKTFCNAbgL7ar9jBhbqPd4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17542278-1730440021-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-01T06:47:11Z},{Key=Name,Value=teamcity-17542278-1730440021-44-n4cpu32-0002}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17542278-1730440021-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-01T06:47:11Z},{Key=Name,Value=teamcity-17542278-1730440021-44-n4cpu32-0002}] --user-data file:///tmp/aws-startup-script2357442709 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping831876523 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test ycsb/E/nodes=3/cpu=32 failed: (test_runner.go:816).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-qFqjpTKTFCNAbgL7ar9jBhbqPd4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-01T07:06:29Z},{Key=Name,Value=teamcity-17542278-1730440021-48-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17542278-1730440021-48-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-11-01T07:06:29Z},{Key=Name,Value=teamcity-17542278-1730440021-48-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17542278-1730440021-48-n4cpu32}] --user-data file:///tmp/aws-startup-script1424727446 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2225351146 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test cdc/tpcc-1000/sink=kafka failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,arch=arm64,created=2024-11-01t07_41_22z,roachprod=true,cluster=teamcity-17542277-1730440131-39-n4cpu16,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-4 --metadata-from-file startup-script=/tmp/gce-startup-script3501621168 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-f teamcity-17542277-1730440131-39-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-c,
      us-east1-b zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-c,
      us-east1-b zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test sysbench/oltp_insert/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test costfuzz/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: T2A instances are not supported outside of [asia-southeast1-b,asia-southeast1-c,europe-west4-a,europe-west4-b,europe-west4-c,us-central1-a,us-central1-b,us-central1-f] [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test sysbench/oltp_read_write/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-c,
      us-east1-b zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Consider trying your request in the us-east1-c,
      us-east1-b zone(s), which currently has capacity to accommodate your request.
      Alternatively, you can try your request again with a different VM hardware configuration
      or at a later time. For more information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: us-east1-c,us-east1-b
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-Je638zVwQT3aRvi9l-byK6e32x8= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-11-06T07:16:30Z},{Key=Name,Value=teamcity-17613258-1730875777-23-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17613258-1730875777-23-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-11-06T07:16:30Z},{Key=Name,Value=teamcity-17613258-1730875777-23-n4cpu32-0003},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17613258-1730875777-23-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script3748709622 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2388569747 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test kv95/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ ee625f92aaca9a68ff53706626b2c5050a91a6f4:

test sysbench/oltp_update_non_index/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ c56ba8cd7e0eee749f0a1aa0aee3da8200688613:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ c56ba8cd7e0eee749f0a1aa0aee3da8200688613:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ c56ba8cd7e0eee749f0a1aa0aee3da8200688613:

test sysbench/oltp_read_only/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ c56ba8cd7e0eee749f0a1aa0aee3da8200688613:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17650369-1731048616-115-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-08t12_26_44z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script4196470787 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17650369-1731048616-115-n4cpu96-0001 teamcity-17650369-1731048616-115-n4cpu96-0002 teamcity-17650369-1731048616-115-n4cpu96-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test alterpk-tpcc-250 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test failover/non-system/pause/lease=expiration failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-09t09_37_47z,roachprod=true,cluster=teamcity-17671423-1731134977-60-n7cpu2,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2001242829 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17671423-1731134977-60-n7cpu2-0001 teamcity-17671423-1731134977-60-n7cpu2-0002 teamcity-17671423-1731134977-60-n7cpu2-0003 teamcity-17671423-1731134977-60-n7cpu2-0004 teamcity-17671423-1731134977-60-n7cpu2-0005 teamcity-17671423-1731134977-60-n7cpu2-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test failover/partial/lease-leader failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,arch=amd64,created=2024-11-09t09_41_08z,roachprod=true,cluster=teamcity-17671423-1731134977-64-n7cpu2,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-2 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2150274144 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17671423-1731134977-64-n7cpu2-0001 teamcity-17671423-1731134977-64-n7cpu2-0002 teamcity-17671423-1731134977-64-n7cpu2-0003 teamcity-17671423-1731134977-64-n7cpu2-0004 teamcity-17671423-1731134977-64-n7cpu2-0005 teamcity-17671423-1731134977-64-n7cpu2-0006]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=2
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test import/tpcc/warehouses=1000/nodes=32 failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,created=2024-11-09t09_56_58z,roachprod=true,cluster=teamcity-17671423-1731134977-81-n32cpu4,lifetime=12h0m0s,arch=amd64 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3201018540 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17671423-1731134977-81-n32cpu4-0001 teamcity-17671423-1731134977-81-n32cpu4-0002 teamcity-17671423-1731134977-81-n32cpu4-0003 teamcity-17671423-1731134977-81-n32cpu4-0004 teamcity-17671423-1731134977-81-n32cpu4-0005 teamcity-17671423-1731134977-81-n32cpu4-0006 teamcity-17671423-1731134977-81-n32cpu4-0007 teamcity-17671423-1731134977-81-n32cpu4-0008 teamcity-17671423-1731134977-81-n32cpu4-0009 teamcity-17671423-1731134977-81-n32cpu4-0010 teamcity-17671423-1731134977-81-n32cpu4-0011 teamcity-17671423-1731134977-81-n32cpu4-0012 teamcity-17671423-1731134977-81-n32cpu4-0013 teamcity-17671423-1731134977-81-n32cpu4-0014 teamcity-17671423-1731134977-81-n32cpu4-0015 teamcity-17671423-1731134977-81-n32cpu4-0016 teamcity-17671423-1731134977-81-n32cpu4-0017 teamcity-17671423-1731134977-81-n32cpu4-0018 teamcity-17671423-1731134977-81-n32cpu4-0019 teamcity-17671423-1731134977-81-n32cpu4-0020 teamcity-17671423-1731134977-81-n32cpu4-0021 teamcity-17671423-1731134977-81-n32cpu4-0022 teamcity-17671423-1731134977-81-n32cpu4-0023 teamcity-17671423-1731134977-81-n32cpu4-0024 teamcity-17671423-1731134977-81-n32cpu4-0025 teamcity-17671423-1731134977-81-n32cpu4-0026 teamcity-17671423-1731134977-81-n32cpu4-0027 teamcity-17671423-1731134977-81-n32cpu4-0028 teamcity-17671423-1731134977-81-n32cpu4-0029 teamcity-17671423-1731134977-81-n32cpu4-0030 teamcity-17671423-1731134977-81-n32cpu4-0031 teamcity-17671423-1731134977-81-n32cpu4-0032]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0004].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0005].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0012].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0013].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0015].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0019].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0020].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0021].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0024].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0025].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0026].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0027].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0028].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0030].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17671423-1731134977-81-n32cpu4-0032].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv95limited-spans/enc=false/nodes=1/cpu=32/seq/no-load-splitting failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv0/enc=false/nodes=1/cpu=32 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv0/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv95/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-4 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17681622-1731221323-118-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-10t11_01_04z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script1077348030 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17681622-1731221323-118-n4cpu96-0001 teamcity-17681622-1731221323-118-n4cpu96-0002 teamcity-17681622-1731221323-118-n4cpu96-0003]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-d/instances/teamcity-17681622-1731221323-118-n4cpu96-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17681622-1731221323-131-n4cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-10t11_09_11z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script3507939554 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17681622-1731221323-131-n4cpu96-0001 teamcity-17681622-1731221323-131-n4cpu96-0002 teamcity-17681622-1731221323-131-n4cpu96-0003]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17681622-1731221323-131-n4cpu96-0001].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17681622-1731221323-162-n5cpu96,lifetime=12h0m0s,arch=amd64,created=2024-11-10t13_05_33z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --local-ssd interface=NVME --machine-type n2d-custom-96-196608 --min-cpu-platform AMD Milan --metadata-from-file startup-script=/tmp/gce-startup-script1543124291 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17681622-1731221323-162-n5cpu96-0001 teamcity-17681622-1731221323-162-n5cpu96-0002 teamcity-17681622-1731221323-162-n5cpu96-0003 teamcity-17681622-1731221323-162-n5cpu96-0004]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17681622-1731221323-162-n5cpu96-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17681622-1731221323-162-n5cpu96-0003].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-c/instances/teamcity-17681622-1731221323-162-n5cpu96-0004].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'LOCAL_SSD_TOTAL_GB_PER_VM_FAMILY' exceeded.  Limit: 80000.0 in region us-east1.
    metric name = compute.googleapis.com/local_ssd_total_storage_per_vm_family
    limit name = LOCAL-SSD-TOTAL-GB-PER-VM-FAMILY-per-project-region
    limit = 80000.0
    dimensions = region: us-east1
vm_family: N2D
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test sysbench/oltp_point_select/nodes=3/cpu=32/conc=256 failed: (test_runner.go:816).func4: in provider: gce: machine type t2a-standard-16 is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ 2e9644dd47ffff1befda9f8f6051d85e16bf4182:

test unoptimized-query-oracle/disable-rules=all/rand-multi-region failed: (test_runner.go:816).func4: in provider: gce: T2A instances are not supported outside of [asia-southeast1-b,asia-southeast1-c,europe-west4-a,europe-west4-b,europe-west4-c,us-central1-a,us-central1-b,us-central1-f] [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:816).func4: in provider: aws: machine type m7gd.xlarge is arm64, but requested arch is amd64 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test backup-restore/online-restore failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17711515-1731394259-21-n4cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-12t07_22_18z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script858380095 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711515-1731394259-21-n4cpu4-0001 teamcity-17711515-1731394259-21-n4cpu4-0002 teamcity-17711515-1731394259-21-n4cpu4-0003]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711515-1731394259-21-n4cpu4-0002].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711515-1731394259-21-n4cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test c2c/mixed-version failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17711515-1731394259-25-n9cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-12t07_30_15z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script2688696560 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711515-1731394259-25-n9cpu4-0001 teamcity-17711515-1731394259-25-n9cpu4-0002 teamcity-17711515-1731394259-25-n9cpu4-0003 teamcity-17711515-1731394259-25-n9cpu4-0004 teamcity-17711515-1731394259-25-n9cpu4-0005 teamcity-17711515-1731394259-25-n9cpu4-0006 teamcity-17711515-1731394259-25-n9cpu4-0007 teamcity-17711515-1731394259-25-n9cpu4-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test c2c/shutdown/src/coordinator failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,arch=amd64,created=2024-11-12t07_32_29z,roachprod=true,cluster=teamcity-17711515-1731394259-28-n9cpu8,lifetime=12h0m0s --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-8 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3253766797 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17711515-1731394259-28-n9cpu8-0001 teamcity-17711515-1731394259-28-n9cpu8-0002 teamcity-17711515-1731394259-28-n9cpu8-0003 teamcity-17711515-1731394259-28-n9cpu8-0004 teamcity-17711515-1731394259-28-n9cpu8-0005 teamcity-17711515-1731394259-28-n9cpu8-0006 teamcity-17711515-1731394259-28-n9cpu8-0007 teamcity-17711515-1731394259-28-n9cpu8-0008]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test cdc/kafka-chaos-single-row failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels spot=true,usage=roachtest,roachprod=true,cluster=teamcity-17711515-1731394259-39-n4cpu16,lifetime=12h0m0s,arch=amd64,created=2024-11-12t08_11_44z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --provisioning-model SPOT --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3840019397 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-c teamcity-17711515-1731394259-39-n4cpu16-0004]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test copy/bank/rows=100000,nodes=5,txn=true failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17711515-1731394259-57-n5cpu4,lifetime=12h0m0s,arch=amd64,created=2024-11-12t08_58_33z --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --local-ssd interface=NVME --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script928924726 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-b teamcity-17711515-1731394259-57-n5cpu4-0001 teamcity-17711515-1731394259-57-n5cpu4-0002 teamcity-17711515-1731394259-57-n5cpu4-0003 teamcity-17711515-1731394259-57-n5cpu4-0004 teamcity-17711515-1731394259-57-n5cpu4-0005]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711515-1731394259-57-n5cpu4-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-east1-b/instances/teamcity-17711515-1731394259-57-n5cpu4-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.cluster_creation failed with artifacts on release-24.2.5-rc @ beddec16a5a438d66bd3f5f4202085cf21c0973c:

test disk-stalled/detection/cgroup/read-write/logs-too=true failed: (test_runner.go:816).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,lifetime=12h0m0s,arch=amd64,created=2024-11-12t09_01_46z,roachprod=true,cluster=teamcity-17711515-1731394259-63-n4cpu4 --scopes cloud-platform --image ubuntu-2204-jammy-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type n2-standard-4 --min-cpu-platform Intel Ice Lake --metadata-from-file startup-script=/tmp/gce-startup-script3778377629 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-east1-d teamcity-17711515-1731394259-63-n4cpu4-0001 teamcity-17711515-1731394259-63-n4cpu4-0002 teamcity-17711515-1731394259-63-n4cpu4-0003]
Output: ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'IN_USE_ADDRESSES' exceeded.  Limit: 1024.0 in region us-east1.
    metric name = compute.googleapis.com/regional_in_use_addresses
    limit name = IN-USE-ADDRESSES-per-project-region
    limit = 1024.0
    dimensions = region: us-east1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #134925 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3.0-rc] - #134027 roachtest: cluster_creation failed [O-roachtest O-robot X-infra-flake branch-release-23.1.29-rc] - #133805 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.15-rc] - #133788 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.7-rc] - #132974 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.3] - #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #133780