roachtest: cluster_creation failed

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv0/enc=false/nodes=3/cpu=32 failed: (test_runner.go:821).func4: in provider: aws: failed to run: aws ec2 run-instances --associate-public-ip-address --count 1 --instance-type c6id.8xlarge --image-id ami-0e83be366243f524a --key-name teamcity-runner-trpPqh6FOv8HihB2VKAw4r4iv44= --region us-east-2 --security-group-ids sg-04d72b57e29d671f1 --subnet-id subnet-0258dc9d1b6473d84 --tag-specifications ResourceType=instance,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17162325-1728020899-20-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-04T06:12:00Z},{Key=Name,Value=teamcity-17162325-1728020899-20-n4cpu32-0003},{Key=Roachprod,Value=true}] ResourceType=volume,Tags=[{Key=usage,Value=roachtest},{Key=Cluster,Value=teamcity-17162325-1728020899-20-n4cpu32},{Key=Lifetime,Value=12h0m0s},{Key=arch,Value=amd64},{Key=Created,Value=2024-10-04T06:12:00Z},{Key=Name,Value=teamcity-17162325-1728020899-20-n4cpu32-0003},{Key=Roachprod,Value=true}] --user-data file:///tmp/aws-startup-script2197376261 --iam-instance-profile Name=roachprod-testing --block-device-mapping file:///tmp/aws-block-device-mapping2219771187 --output json: stderr: 
An error occurred (InsufficientInstanceCapacity) when calling the RunInstances operation (reached max retries: 2): We currently do not have sufficient c6id.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get c6id.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b, us-east-2c.: exit status 254 [owner=test-eng]

Parameters:

• ROACHTEST_cloud=aws
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-42737

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank-multitable/majority-ring-subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-160-n6cpu4-0006: creating IP teamcity-17202433-1728280076-160-n6cpu4-0006: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank-multitable/strobe-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-171-n6cpu4-0004: creating IP teamcity-17202433-1728280076-171-n6cpu4-0004: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test hibernate-spatial failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-162-n1cpu4-0001: creating IP teamcity-17202433-1728280076-162-n1cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test inconsistency failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-163-n3cpu4-0002: creating IP teamcity-17202433-1728280076-163-n3cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank-multitable/subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-174-n6cpu4-0005: creating IP teamcity-17202433-1728280076-174-n6cpu4-0005: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank-multitable/start-stop-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-178-n6cpu4-0004: creating IP teamcity-17202433-1728280076-178-n6cpu4-0004: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank/strobe-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-190-n6cpu4-0002: creating IP teamcity-17202433-1728280076-190-n6cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank/start-stop-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-187-n6cpu4-0001: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDPDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 1200, Current Usage: 1200, Additional Required: 4, (Minimum) New Limit Required: 1204. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDPDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:1204,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDPDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank/subcritical-skews-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-194-n6cpu4-0001: creating IP teamcity-17202433-1728280076-194-n6cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/bank/majority-ring failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-193-n6cpu4-0006: creating IP teamcity-17202433-1728280076-193-n6cpu4-0006: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/g2/majority-ring-subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: resources.GroupsClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/monotonic/split failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-199-n6cpu4-0002: creating IP teamcity-17202433-1728280076-199-n6cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/g2/split failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksCreateOrUpdateFuture#Result: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/monotonic/start-stop-2 failed: (test_runner.go:821).func4: in provider: azure: resources.GroupsClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/g2/majority-ring failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksCreateOrUpdateFuture#Result: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/monotonic/subcritical-skews-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-211-n6cpu4-0003: creating IP teamcity-17202433-1728280076-211-n6cpu4-0003: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/monotonic/majority-ring-subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-208-n6cpu4-0002: creating IP teamcity-17202433-1728280076-208-n6cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/monotonic/majority-ring failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-205-n6cpu4-0001: creating IP teamcity-17202433-1728280076-205-n6cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/majority-ring-subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-221-n6cpu4-0003: creating IP teamcity-17202433-1728280076-221-n6cpu4-0003: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/parts-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-219-n6cpu4-0006: creating IP teamcity-17202433-1728280076-219-n6cpu4-0006: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/strobe-skews failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="RetryableError" Message="A retryable error occurred." Details=[{"code":"RetryableErrorDueToAnotherOperation","message":"Operation PutVirtualNetworkOperation (93fcc151-90a4-4ed9-9f6a-ad600aa43072) is updating resource /subscriptions/73075e4f-150d-4f4f-956b-053507b61fa7/resourceGroups/roachprod-vnets-eastus/providers/Microsoft.Network/virtualNetworks/roachprod-vnets-eastus. The call can be retried in 14 seconds."}] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/majority-ring-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-225-n6cpu4-0004: creating IP teamcity-17202433-1728280076-225-n6cpu4-0004: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/subcritical-skews-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-230-n6cpu4-0006: creating IP teamcity-17202433-1728280076-230-n6cpu4-0006: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/strobe-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-237-n6cpu4-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 1400, Current Usage: 1400, Additional Required: 4, (Minimum) New Limit Required: 1404. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:1404,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/split failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-243-n6cpu4-0005: creating IP teamcity-17202433-1728280076-243-n6cpu4-0005: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/majority-ring failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksCreateOrUpdateFuture#Result: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/register/subcritical-skews-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-250-n6cpu4-0001: creating IP teamcity-17202433-1728280076-250-n6cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/multi-register/subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-244-n6cpu4-0002: creating IP teamcity-17202433-1728280076-244-n6cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/register/split failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-253-n6cpu4-0006: creating IP teamcity-17202433-1728280076-253-n6cpu4-0006: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sets/parts-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-248-n6cpu4-0002: creating IP teamcity-17202433-1728280076-248-n6cpu4-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/subcritical-skews-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-261-n6cpu4-0001: creating IP teamcity-17202433-1728280076-261-n6cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/parts-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-249-n6cpu4-0005: creating IP teamcity-17202433-1728280076-249-n6cpu4-0005: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sequential/majority-ring-subcritical-skews failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-256-n6cpu4-0005: creating IP teamcity-17202433-1728280076-256-n6cpu4-0005: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test knex failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-258-n1cpu4-0001: creating IP teamcity-17202433-1728280076-258-n1cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/sets/start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: resources.GroupsClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv0/enc=false/nodes=3/size=64kb failed: (test_runner.go:821).func4: in provider: azure: resources.GroupsClient#Get: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test jepsen/register/majority-ring-start-kill-2 failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksCreateOrUpdateFuture#Result: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv0/enc=false/nodes=3/lease=expiration failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-283-n4cpu8-0002: creating IP teamcity-17202433-1728280076-283-n4cpu8-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv50/enc=false/nodes=4/cpu=96/batch=64 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-267-n5cpu96-0001: creating IP teamcity-17202433-1728280076-267-n5cpu96-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=96
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv50/rangelookups/split/nodes=8 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-288-n9cpu8-0004: creating IP teamcity-17202433-1728280076-288-n9cpu8-0004: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=true
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=false/nodes=4 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-294-n5cpu8-0002: creating IP teamcity-17202433-1728280076-294-n5cpu8-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=false/nodes=1/cpu=32/mt-shared-process failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-286-n2cpu32-0001: creating IP teamcity-17202433-1728280076-286-n2cpu32-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=32
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test liquibase failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-287-n1cpu4-0001: creating IP teamcity-17202433-1728280076-287-n1cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=false/nodes=3/mt-shared-process failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-291-n4cpu8-0002: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 1400, Current Usage: 1400, Additional Required: 8, (Minimum) New Limit Required: 1408. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:1408,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test loqrecovery/workload=movr/rangeSize=2mb failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-310-n6cpu4-0001: creating IP teamcity-17202433-1728280076-310-n6cpu4-0001: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=true/nodes=3 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-299-n4cpu8-0003: compute.VirtualMachinesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: autorest/azure: Service returned an error. Status=<nil> Code="OperationNotAllowed" Message="Operation could not be completed as it results in exceeding approved standardDDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: eastus, Current Limit: 1400, Current Usage: 1400, Additional Required: 8, (Minimum) New Limit Required: 1408. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%2273075e4f-150d-4f4f-956b-053507b61fa7%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22eastus%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:1408,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests" [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test loqrecovery/half-online/workload=tpcc/rangeSize=default failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksCreateOrUpdateFuture#Result: Failure responding to request: StatusCode=429 -- Original Error: autorest/azure: Service returned an error. Status=429 Code="SubscriptionRequestsThrottled" Message="Number of 'read' requests for subscription '73075e4f-150d-4f4f-956b-053507b61fa7' actor 'c684e43c-0da3-4702-8ba8-ea9eba46c47f' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information." [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test loqrecovery/half-online/workload=movr/rangeSize=default failed: (test_runner.go:821).func4: in provider: azure: for location "eastus": creating Azure VNet "common" in "roachprod-vnets-eastus": network.VirtualNetworksClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="RetryableError" Message="A retryable error occurred." Details=[{"code":"RetryableErrorDueToAnotherOperation","message":"Operation PutVirtualNetworkOperation (fca74f5b-840d-4e17-887b-c1573faff762) is updating resource /subscriptions/73075e4f-150d-4f4f-956b-053507b61fa7/resourceGroups/roachprod-vnets-eastus/providers/Microsoft.Network/virtualNetworks/roachprod-vnets-eastus. The call can be retried in 12 seconds."}] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=false/nodes=3/batch=16 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-308-n4cpu8-0002: creating IP teamcity-17202433-1728280076-308-n4cpu8-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=8
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.cluster_creation failed with artifacts on release-24.2.4-rc @ 8140b8e294fb16a293d0aae662bcf5c1d488c852:

test kv95/enc=false/nodes=3/cpu=96 failed: (test_runner.go:821).func4: in provider: azure: creating VM teamcity-17202433-1728280076-315-n4cpu96-0002: creating IP teamcity-17202433-1728280076-315-n4cpu96-0002: network.PublicIPAddressesClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="PublicIPCountLimitReached" Message="Cannot create more than 1000 public IP addresses for this subscription in this region." Details=[] [owner=test-eng]

Parameters:

• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=96
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Same failure on other branches

- #131730 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1.6-rc] - #131729 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1.28-rc] - #131725 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2.13-rc] - #126981 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.2] - #123745 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-master] - #121332 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-24.1] - #112176 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.2] - #108533 roachtest: cluster_creation failed [O-roachtest O-robot T-testeng X-infra-flake branch-release-23.1]

This test on roachdash | Improve this report!