roachtest: cluster_creation failed

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test admission-control/disk-bandwidth-limiter failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:14:34Z},{Key=Name,Value=teamcity-17376626-1729317016-19-n2cpu8-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:14:34Z},{Key=Name,Value=teamcity-17376626-1729317016-19-n2cpu8-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script863458723 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1321626091 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-43373

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-35-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T06:36:12Z},{Key=Name,Value=teamcity-17376626-1729317016-35-n2cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-35-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T06:36:12Z},{Key=Name,Value=teamcity-17376626-1729317016-35-n2cpu32-0001}] --user-data file:///tmp/aws-startup-script1846387131 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3311457935 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17376626-1729317016-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:49:11Z},{Key=Name,Value=teamcity-17376626-1729317016-44-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17376626-1729317016-44-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:49:11Z},{Key=Name,Value=teamcity-17376626-1729317016-44-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script944005610 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3524221112 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-47-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:51:37Z},{Key=Name,Value=teamcity-17376626-1729317016-47-n4cpu32-0003},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-47-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:51:37Z},{Key=Name,Value=teamcity-17376626-1729317016-47-n4cpu32-0003},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script3918861526 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping753787574 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:54:08Z},{Key=Name,Value=teamcity-17376626-1729317016-50-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-50-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:54:08Z},{Key=Name,Value=teamcity-17376626-1729317016-50-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2979310631 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping65384041 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-53-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-53-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:56:42Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-53-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-53-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T06:56:42Z}] --user-data file:///tmp/aws-startup-script1272277499 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping822618068 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test kv0/enc=false/nodes=3/cpu=96 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-60-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T07:06:32Z},{Key=Name,Value=teamcity-17376626-1729317016-60-n4cpu96-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-60-n4cpu96},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T07:06:32Z},{Key=Name,Value=teamcity-17376626-1729317016-60-n4cpu96-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1605527747 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3256357657 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/C/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-102-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-102-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T08:48:08Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-102-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-102-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T08:48:08Z}] --user-data file:///tmp/aws-startup-script133431599 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1822303133 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/D/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.2xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-105-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T08:50:11Z},{Key=Name,Value=teamcity-17376626-1729317016-105-n4cpu8-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-105-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T08:50:11Z},{Key=Name,Value=teamcity-17376626-1729317016-105-n4cpu8-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1585319292 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1249018149 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/E/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.2xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T08:52:17Z},{Key=Name,Value=teamcity-17376626-1729317016-108-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-108-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T08:52:17Z},{Key=Name,Value=teamcity-17376626-1729317016-108-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-108-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3526464649 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1110651258 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T08:54:15Z},{Key=Name,Value=teamcity-17376626-1729317016-111-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-111-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T08:54:15Z},{Key=Name,Value=teamcity-17376626-1729317016-111-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-111-n4cpu32}] --user-data file:///tmp/aws-startup-script2145886442 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3559384915 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-116-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:03:31Z},{Key=Name,Value=teamcity-17376626-1729317016-116-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-116-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:03:31Z},{Key=Name,Value=teamcity-17376626-1729317016-116-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script741263460 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping769460506 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/B/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-119-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T09:05:27Z},{Key=Name,Value=teamcity-17376626-1729317016-119-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-119-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-19T09:05:27Z},{Key=Name,Value=teamcity-17376626-1729317016-119-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script3749961565 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1911015822 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/B/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-122-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:07:30Z},{Key=Name,Value=teamcity-17376626-1729317016-122-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-122-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:07:30Z},{Key=Name,Value=teamcity-17376626-1729317016-122-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2413867081 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2844064076 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/B/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17376626-1729317016-125-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:09:30Z},{Key=Name,Value=teamcity-17376626-1729317016-125-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17376626-1729317016-125-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:09:30Z},{Key=Name,Value=teamcity-17376626-1729317016-125-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script481640704 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1294674589 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/C/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:11:28Z},{Key=Name,Value=teamcity-17376626-1729317016-128-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-128-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:11:28Z},{Key=Name,Value=teamcity-17376626-1729317016-128-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-128-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script4030319641 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1324966844 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/C/nodes=3/cpu=32/isolation-level=read-committed failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-131-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-131-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:13:29Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17376626-1729317016-131-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-131-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:13:29Z}] --user-data file:///tmp/aws-startup-script568479536 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3571715384 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/C/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-134-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:15:25Z},{Key=Name,Value=teamcity-17376626-1729317016-134-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-134-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:15:25Z},{Key=Name,Value=teamcity-17376626-1729317016-134-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2204486021 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping827693414 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test ycsb/C/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-6XkJAABmvqLSI84HVCoCM0l7voI= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-137-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:17:35Z},{Key=Name,Value=teamcity-17376626-1729317016-137-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17376626-1729317016-137-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-19T09:17:35Z},{Key=Name,Value=teamcity-17376626-1729317016-137-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script2284718122 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2488236779 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

test cdc/cloud-sink-gcs/assume-role failed: (test_runner.go:846).func4: in provider: gce: Command: gcloud [compute instances create --subnet default --labels usage=roachtest,roachprod=true,cluster=teamcity-17376625-1729316924-51-n4cpu16,lifetime=12h0m0s,arch=arm64,created=2024-10-19t09_45_52z --scopes cloud-platform --image ubuntu-2204-jammy-arm64-v20240319 --image-project ubuntu-os-cloud --boot-disk-type pd-ssd --service-account 21965078311-compute@developer.gserviceaccount.com --maintenance-policy MIGRATE --create-disk type=pd-ssd,size=500GB,auto-delete=yes --machine-type t2a-standard-16 --metadata-from-file startup-script=/tmp/gce-startup-script1973961309 --project cockroach-ephemeral --boot-disk-size=32GB --zone us-central1-a teamcity-17376625-1729316924-51-n4cpu16-0001 teamcity-17376625-1729316924-51-n4cpu16-0002 teamcity-17376625-1729316924-51-n4cpu16-0003]
Output: Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17376625-1729316924-51-n4cpu16-0001].
Created [https://www.googleapis.com/compute/v1/projects/cockroach-ephemeral/zones/us-central1-a/instances/teamcity-17376625-1729316924-51-n4cpu16-0003].
WARNING: Some requests generated warnings:
 - Disk size: '32 GB' is larger than image size: '10 GB'. You might need to resize the root repartition manually if the operating system does not support automatic resizing. See https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_pd for details.
 - The resource 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240319' is deprecated. A suggested replacement is 'projects/ubuntu-os-cloud/global/images/ubuntu-2204-jammy-arm64-v20240927'.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Quota 'T2A_CPUS' exceeded.  Limit: 600.0 in region us-central1.
    metric name = compute.googleapis.com/t2a_cpus
    limit name = T2A-CPUS-per-project-region
    limit = 600.0
    dimensions = region: us-central1
Try your request in another zone, or view documentation on how to increase quotas: https://cloud.google.com/compute/quotas.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test admission-control/disk-bandwidth-limiter failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-20T06:13:11Z},{Key=Name,Value=teamcity-17383450-1729403366-19-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-20T06:13:11Z},{Key=Name,Value=teamcity-17383450-1729403366-19-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-19-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script3545037171 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3828280530 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=1/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-34-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T06:32:22Z},{Key=Name,Value=teamcity-17383450-1729403366-34-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-34-n2cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T06:32:22Z},{Key=Name,Value=teamcity-17383450-1729403366-34-n2cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script1751352121 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1041287252 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T06:47:24Z},{Key=Name,Value=teamcity-17383450-1729403366-46-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-46-n4cpu32}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T06:47:24Z},{Key=Name,Value=teamcity-17383450-1729403366-46-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-46-n4cpu32}] --user-data file:///tmp/aws-startup-script793428852 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4294678426 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-49-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:49:32Z},{Key=Name,Value=teamcity-17383450-1729403366-49-n4cpu32-0003},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-49-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:49:32Z},{Key=Name,Value=teamcity-17383450-1729403366-49-n4cpu32-0003},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script3376971651 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping601543818 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17383450-1729403366-52-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-52-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:51:39Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17383450-1729403366-52-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-52-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:51:39Z}] --user-data file:///tmp/aws-startup-script3479275610 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1091965521 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-55-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:53:50Z},{Key=Name,Value=teamcity-17383450-1729403366-55-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-55-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T06:53:50Z},{Key=Name,Value=teamcity-17383450-1729403366-55-n4cpu32-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2495162147 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping600519349 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv0/enc=false/nodes=3/size=64kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T07:03:24Z},{Key=Name,Value=teamcity-17383450-1729403366-63-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-63-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-20T07:03:24Z},{Key=Name,Value=teamcity-17383450-1729403366-63-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-63-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3323270491 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping283885330 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test kv95/enc=false/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-20T07:36:14Z},{Key=Name,Value=teamcity-17383450-1729403366-70-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-70-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-20T07:36:14Z},{Key=Name,Value=teamcity-17383450-1729403366-70-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-70-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64}] --user-data file:///tmp/aws-startup-script743109976 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1127710580 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test ycsb/A/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-112-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T09:20:33Z},{Key=Name,Value=teamcity-17383450-1729403366-112-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17383450-1729403366-112-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T09:20:33Z},{Key=Name,Value=teamcity-17383450-1729403366-112-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script925306345 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping319667003 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-115-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T09:22:32Z},{Key=Name,Value=teamcity-17383450-1729403366-115-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-115-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T09:22:32Z},{Key=Name,Value=teamcity-17383450-1729403366-115-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script838265068 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1286417519 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

test ycsb/D/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-h9w8sKlx0R_fLfBUE_ONbGhhFQ4= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-121-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T10:23:17Z},{Key=Name,Value=teamcity-17383450-1729403366-121-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17383450-1729403366-121-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-20T10:23:17Z},{Key=Name,Value=teamcity-17383450-1729403366-121-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script2419366597 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2178317880 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.
---
code: ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
errorDetails:
- help:
    links:
    - description: Troubleshooting documentation
      url: https://cloud.google.com/compute/docs/resource-error
- localizedMessage:
    locale: en-US
    message: A n2d-custom-96-196608 VM instance with 16 local SSD(s) is currently
      unavailable in the us-east1-d zone. Alternatively, you can try your request
      again with a different VM hardware configuration or at a later time. For more
      information, see the troubleshooting documentation.
- errorInfo:
    domain: compute.googleapis.com
    metadatas:
      attachment: local-ssd:16
      vmType: n2d-custom-96-196608
      zone: us-east1-d
      zonesAvailable: ''
    reason: resource_availability
message: The zone 'projects/cockroach-ephemeral/zones/us-east1-d' does not have enough
  resources available to fulfill the request.  '(resource type:compute)'.: exit status 1 [owner=test-eng]

Parameters:

ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test admission-control/disk-bandwidth-limiter failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T06:15:44Z},{Key=Name,Value=teamcity-17390168-1729489853-19-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-19-n2cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T06:15:44Z},{Key=Name,Value=teamcity-17390168-1729489853-19-n2cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-19-n2cpu8}] --user-data file:///tmp/aws-startup-script1003860781 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping858529416 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv0/enc=false/nodes=1 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T06:34:05Z},{Key=Name,Value=teamcity-17390168-1729489853-33-n2cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-33-n2cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T06:34:05Z},{Key=Name,Value=teamcity-17390168-1729489853-33-n2cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-33-n2cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script341173409 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2835304252 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m6id.xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m6id.xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv0/enc=false/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-45-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T06:48:27Z},{Key=Name,Value=teamcity-17390168-1729489853-45-n4cpu32-0001}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-45-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T06:48:27Z},{Key=Name,Value=teamcity-17390168-1729489853-45-n4cpu32-0001}] --user-data file:///tmp/aws-startup-script2392883916 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4226201602 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv0/enc=true/nodes=1 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-57-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:09:42Z},{Key=Name,Value=teamcity-17390168-1729489853-57-n2cpu8-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-57-n2cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:09:42Z},{Key=Name,Value=teamcity-17390168-1729489853-57-n2cpu8-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script859628411 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1626405559 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient m7gd.2xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get m7gd.2xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv0/enc=true/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-60-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:11:31Z},{Key=Name,Value=teamcity-17390168-1729489853-60-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-60-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:11:31Z},{Key=Name,Value=teamcity-17390168-1729489853-60-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3913972008 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2692621801 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/cpu=32/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:26:23Z},{Key=Name,Value=teamcity-17390168-1729489853-69-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-69-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:26:23Z},{Key=Name,Value=teamcity-17390168-1729489853-69-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-69-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3976632549 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1992848875 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/cpu=32/seq failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17390168-1729489853-73-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-73-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:29:45Z}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Name,Value=teamcity-17390168-1729489853-73-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-73-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:29:45Z}] --user-data file:///tmp/aws-startup-script3762973930 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3324008080 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/cpu=32/size=4kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-76-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:31:35Z},{Key=Name,Value=teamcity-17390168-1729489853-76-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-76-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:31:35Z},{Key=Name,Value=teamcity-17390168-1729489853-76-n4cpu32-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script3588599504 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1357172861 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/cpu=32/size=64kb failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-79-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:33:34Z},{Key=Name,Value=teamcity-17390168-1729489853-79-n4cpu32-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-79-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:33:34Z},{Key=Name,Value=teamcity-17390168-1729489853-79-n4cpu32-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1773270562 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping4199859985 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:35:29Z},{Key=Name,Value=teamcity-17390168-1729489853-82-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-82-n4cpu8},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:35:29Z},{Key=Name,Value=teamcity-17390168-1729489853-82-n4cpu8-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-82-n4cpu8},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script3520241225 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2806437798 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/tracing failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-86-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:39:33Z},{Key=Name,Value=teamcity-17390168-1729489853-86-n4cpu8-0002},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-86-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:39:33Z},{Key=Name,Value=teamcity-17390168-1729489853-86-n4cpu8-0002},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script3528852113 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3949571217 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=4 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:41:33Z},{Key=Name,Value=teamcity-17390168-1729489853-89-n5cpu8-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-89-n5cpu8}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T07:41:33Z},{Key=Name,Value=teamcity-17390168-1729489853-89-n5cpu8-0003},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-89-n5cpu8}] --user-data file:///tmp/aws-startup-script3726282738 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2282692777 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): There is no Spot capacity available that matches your request.: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6a.24xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:56:23Z},{Key=Name,Value=teamcity-17390168-1729489853-101-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-101-n4cpu96}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T07:56:23Z},{Key=Name,Value=teamcity-17390168-1729489853-101-n4cpu96-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-101-n4cpu96}] --user-data file:///tmp/aws-startup-script38101507 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3294567384 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=96
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test tpccbench/nodes=3/cpu=16/mt-shared-process failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m6id.4xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-121-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T08:24:40Z},{Key=Name,Value=teamcity-17390168-1729489853-121-n4cpu16-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-121-n4cpu16},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-21T08:24:40Z},{Key=Name,Value=teamcity-17390168-1729489853-121-n4cpu16-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1892517682 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping602948947 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=16
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test ycsb/A/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:28:44Z},{Key=Name,Value=teamcity-17390168-1729489853-126-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-126-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:28:44Z},{Key=Name,Value=teamcity-17390168-1729489853-126-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-126-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script2080561215 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping515572435 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test ycsb/A/nodes=3/cpu=32/mvcc-range-keys=global failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-134-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:36:31Z},{Key=Name,Value=teamcity-17390168-1729489853-134-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17390168-1729489853-134-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:36:31Z},{Key=Name,Value=teamcity-17390168-1729489853-134-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2209386051 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2923311212 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test ycsb/A/nodes=3/cpu=32/uniform failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-140-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:41:59Z},{Key=Name,Value=teamcity-17390168-1729489853-140-n4cpu32-0001},{Key=Spot,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-140-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:41:59Z},{Key=Name,Value=teamcity-17390168-1729489853-140-n4cpu32-0001},{Key=Spot,Value=true}] --user-data file:///tmp/aws-startup-script1883234374 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2881150512 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test ycsb/F/nodes=3 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type m7gd.2xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-21T08:43:59Z},{Key=Name,Value=teamcity-17390168-1729489853-143-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-143-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Created,Value=2024-10-21T08:43:59Z},{Key=Name,Value=teamcity-17390168-1729489853-143-n4cpu8-0002},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-143-n4cpu8},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=arm64}] --user-data file:///tmp/aws-startup-script2963877763 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping3503592968 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cluster_creation failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

test ycsb/B/nodes=3/cpu=32 failed: (test_runner.go:846).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c7gd.8xlarge --image-id ami-05983a09f7dc1c18f --key-name teamcity-runner-_x3-ZQgmB02lstY4N1Cx1gWCxbM= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:45:48Z},{Key=Name,Value=teamcity-17390168-1729489853-146-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-146-n4cpu32},{Key=Lifetime,Value=12h0m0s}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=arch,Value=arm64},{Key=Created,Value=2024-10-21T08:45:48Z},{Key=Name,Value=teamcity-17390168-1729489853-146-n4cpu32-0001},{Key=Spot,Value=true},{Key=Roachprod,Value=true},{Key=Cluster,Value=teamcity-17390168-1729489853-146-n4cpu32},{Key=Lifetime,Value=12h0m0s}] --user-data file:///tmp/aws-startup-script1512669259 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping1580547593 --instance-market-options MarketType=spot,SpotOptions={SpotInstanceType=one-time,InstanceInterruptionBehavior=terminate} --output json: stderr: 
An error occurred (MaxSpotInstanceCountExceeded) when calling the RunInstances operation: Max spot instance count exceeded: exit status 254 [owner=test-eng]

Parameters:

ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=32
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131888 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2.4-rc] - #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

cockroachdb / cockroach

roachtest: cluster_creation failed #132974