search

Azure-Samples / jmeter-aci-terraform

Scalable cloud load/stress testing pipeline solution with Apache JMeter and Terraform to dynamically provision and destroy the required infrastructure on Azure.
MIT License
119 stars 99 forks source link

SETUP: Run Terraform Apply (target=all) #81

Open axlvlv opened 2 years ago

axlvlv commented 2 years ago

This error does not happen every time, but keep happening

Error: Error creating/updating container group "jmeter-worker16" (Resource Group "jmeter-uksouth"): containerinstance.ContainerGroupsClient#CreateOrUpdate: Failure sending request: StatusCode=409 -- Original Error: Code="ServiceAssociationLinkNotReady" Message="Network is not ready for container group 'jmeter-worker16'. Please try again."

matijavugi commented 2 years ago

Hi, have you figured this out? We are experiencing the same thing.

dsteindo commented 2 years ago

As I understand this issue is caused by Terraform trying to create multiple instances in parallel. Because of this parallelism there can be conflicts (error 409) when a resource (e.g. Network) is accessed the same time.

According to the documentation of Terraform https://www.terraform.io/docs/cli/commands/apply.html it defaults to 10 concurrent operations. Setting parallelism to 1 might be a solution, a very slow solution and unviable if you plan to create 4 or more workers.

There is a page on Terraform describing some retry mechanism but I am not sure how to add it into the project https://www.terraform.io/docs/extend/resources/retries-and-customizable-timeouts.html

dsteindo commented 2 years ago

We noticed that an outdated terraform version is used here https://github.com/Azure-Samples/jmeter-aci-terraform/blob/main/pipelines/azure-pipelines.load-test.yml

The version used is 0.13.2 but as of now the version 1.1.3 as specified here https://www.terraform.io/downloads

changing the version causes lots of other issues but this is something we have to try as creating an bug on terraform with an outdated version is not the way to go.

there is a pull request for that https://github.com/Azure-Samples/jmeter-aci-terraform/pull/84

Edit: updating to the lastest version of terraform does not resolve the ticket, reaching out to the terraform team for help now ...