dacbd commented 2 years ago

was taking a look at: iterative/blog-tpi-bees and my impatience resulted in an error... 🔥

git clone git@github.com:iterative/blog-tpi-bees.git
cd blog-tpi-bees/
vim main.tf
terraform init
terraform apply
# 3 minutes later I decide that is too long and Ctrl-C
terraform destroy
vim main.tf #change to a different region
terraform apply
# ERR

Relevant terraform outputs:

terraform init

``` dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform init Initializing the backend... Initializing provider plugins... - Finding iterative/iterative versions matching ">= 0.9.9"... - Installing iterative/iterative v0.10.9... - Installed iterative/iterative v0.10.9 (self-signed, key ID 910A9EF2910B1E37) Partner and community providers are signed by their developers. If you'd like to know more about provider signing, you can read about it here: https://www.terraform.io/docs/cli/plugins/signing.html Terraform has created a lock file .terraform.lock.hcl to record the provider selections it made above. Include this file in your version control repository so that Terraform can guarantee to make the same selections by default when you run "terraform init" in the future. Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary. ```

1st terraform apply

``` dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform apply 2022-04-26T16:23:06.850-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:23:06.881-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.881-0700 2022-04-26T16:23:06.905-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:23:06.937-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.937-0700 2022-04-26T16:23:06.958-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:06 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:06.958-0700 2022-04-26T16:23:06.961-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:23:06.993-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:06.993-0700 2022-04-26T16:23:07.016-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:07 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:07.016-0700 Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # iterative_task.tpi-examples-basic will be created + resource "iterative_task" "tpi-examples-basic" { + addresses = (known after apply) + cloud = "aws" + disk_size = 30 + events = (known after apply) + id = (known after apply) + image = "ubuntu" + logs = (known after apply) + machine = "t3.small" + name = "tpi-examples-basic " + parallelism = 1 + region = "us-east-2" + script = <<-EOT #!/bin/bash sudo apt update sudo apt install -y python3-pip pip3 install -r requirements.txt python3 src/train.py EOT + spot = 0 + ssh_private_key = (sensitive value) + ssh_public_key = (sensitive value) + status = (known after apply) + timeout = 86400 + storage { + output = "shared" + workdir = "." } } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes 2022-04-26T16:23:10.644-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:23:10.678-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:23:10.678-0700 2022-04-26T16:23:10.702-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:23:10.702-0700 iterative_task.tpi-examples-basic: Creating... 2022-04-26T16:23:10.704-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700 2022-04-26T16:23:10.704-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700 2022-04-26T16:23:10.704-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700 2022-04-26T16:23:10.704-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:23:10 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:23:10.704-0700 TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait. iterative_task.tpi-examples-basic: Still creating... [10s elapsed] iterative_task.tpi-examples-basic: Still creating... [20s elapsed] iterative_task.tpi-examples-basic: Still creating... [30s elapsed] iterative_task.tpi-examples-basic: Still creating... [40s elapsed] iterative_task.tpi-examples-basic: Still creating... [50s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m0s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m10s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m20s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m30s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m40s elapsed] iterative_task.tpi-examples-basic: Still creating... [1m50s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m0s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m10s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m20s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m30s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m40s elapsed] iterative_task.tpi-examples-basic: Still creating... [2m50s elapsed] iterative_task.tpi-examples-basic: Still creating... [3m0s elapsed] ^CStopping operation... Interrupt received. Please wait for Terraform to exit or data loss may occur. Gracefully shutting down... 2022-04-26T16:26:18.289-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_038.png: Failed to copy: RequestCanceled: request context canceled caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700 2022-04-26T16:26:18.289-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_011.png: Failed to copy: RequestCanceled: request context canceled caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700 2022-04-26T16:26:18.289-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_004.png: Failed to copy: RequestCanceled: request context canceled caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700 2022-04-26T16:26:18.289-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:18 ERROR : data/bee_imgs/bee_imgs/016_010.png: Failed to copy: RequestCanceled: request context canceled caused by: context canceled: timestamp=2022-04-26T16:26:18.280-0700 ╷ │ Error: execution halted │ │ ╵ ╷ │ Error: execution halted │ │ ╵ ╷ │ Error: RequestCanceled: request context canceled │ caused by: context canceled │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ ╷ │ Error: operation error Auto Scaling: DeleteAutoScalingGroup, https response error StatusCode: 0, RequestID: , canceled, context canceled │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ ```

terraform destroy

``` dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform destroy 2022-04-26T16:26:23.544-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:26:23.577-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.577-0700 2022-04-26T16:26:23.602-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:26:23.633-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.633-0700 2022-04-26T16:26:23.655-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:23 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:26:23.655-0700 2022-04-26T16:26:23.658-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:26:23.690-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:23.690-0700 iterative_task.tpi-examples-basic: Refreshing state... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry] TPI [INFO] Reading resources... (this may happen several times) TPI [INFO] [1/9] Reading DefaultVPC... TPI [INFO] [2/9] Reading DefaultVPCSubnet... TPI [INFO] [3/9] Reading Image... TPI [INFO] [4/9] Reading Bucket... TPI [INFO] [5/9] Reading SecurityGroup... TPI [INFO] [6/9] Reading KeyPair... TPI [INFO] [7/9] Reading Credentials... TPI [INFO] [8/9] Reading LaunchTemplate... TPI [INFO] [9/9] Reading AutoScalingGroup... TPI [INFO] Read completed TPI [INFO] Reading resources... (this may happen several times) TPI [INFO] [1/9] Reading DefaultVPC... TPI [INFO] [2/9] Reading DefaultVPCSubnet... TPI [INFO] [3/9] Reading Image... TPI [INFO] [4/9] Reading Bucket... TPI [INFO] [5/9] Reading SecurityGroup... TPI [INFO] [6/9] Reading KeyPair... TPI [INFO] [7/9] Reading Credentials... TPI [INFO] [8/9] Reading LaunchTemplate... TPI [INFO] [9/9] Reading AutoScalingGroup... TPI [INFO] Read completed TPI [INFO] Reading resources... (this may happen several times) TPI [INFO] [1/9] Reading DefaultVPC... TPI [INFO] [2/9] Reading DefaultVPCSubnet... TPI [INFO] [3/9] Reading Image... TPI [INFO] [4/9] Reading Bucket... TPI [INFO] [5/9] Reading SecurityGroup... TPI [INFO] [6/9] Reading KeyPair... TPI [INFO] [7/9] Reading Credentials... TPI [INFO] [8/9] Reading LaunchTemplate... TPI [INFO] [9/9] Reading AutoScalingGroup... TPI [INFO] Read completed TPI [INFO] aws t3.small in us-east-2 TPI [INFO] TPI [INFO] Status: queued • 2022-04-26T16:26:33.846-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:26:33 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:26:33.846-0700 2022-04-26T16:26:33.854-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:26:33.893-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:33.893-0700 Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: - destroy Terraform will perform the following actions: # iterative_task.tpi-examples-basic will be destroyed - resource "iterative_task" "tpi-examples-basic" { - addresses = [] -> null - cloud = "aws" -> null - disk_size = 30 -> null - events = [] -> null - id = "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry" -> null - image = "ubuntu" -> null - logs = [] -> null - machine = "t3.small" -> null - name = "tpi-examples-basic " -> null - parallelism = 1 -> null - region = "us-east-2" -> null - script = <<-EOT #!/bin/bash sudo apt update sudo apt install -y python3-pip pip3 install -r requirements.txt python3 src/train.py EOT -> null - spot = 0 -> null - ssh_private_key = (sensitive value) - ssh_public_key = (sensitive value) - status = { - "running" = 0 } -> null - timeout = 86400 -> null - storage { - output = "shared" -> null - workdir = "." -> null } } Plan: 0 to add, 0 to change, 1 to destroy. Do you really want to destroy all resources? Terraform will destroy all your managed infrastructure, as shown above. There is no undo. Only 'yes' will be accepted to confirm. Enter a value: yes 2022-04-26T16:26:37.305-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:26:37.340-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:26:37.338-0700 iterative_task.tpi-examples-basic: Destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry] TPI [INFO] Destruction may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait. iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 10s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 20s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 30s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 40s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 50s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 1m0s elapsed] iterative_task.tpi-examples-basic: Still destroying... [id=tpi-tpi-examples-basic-63n5cpxk-2tqun9ry, 1m10s elapsed] iterative_task.tpi-examples-basic: Destruction complete after 1m14s Destroy complete! Resources: 1 destroyed. ```

2nd terraform apply

``` dabarnes@ubuntu:~/tests/blog-tpi-bees$ terraform apply 2022-04-26T16:28:28.907-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:28:28.942-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:28.942-0700 2022-04-26T16:28:28.965-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:28:28.996-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:28.995-0700 2022-04-26T16:28:29.019-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:29 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:29.019-0700 2022-04-26T16:28:29.022-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:28:29.053-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:29.053-0700 2022-04-26T16:28:29.075-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:29 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:29.075-0700 Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # iterative_task.tpi-examples-basic will be created + resource "iterative_task" "tpi-examples-basic" { + addresses = (known after apply) + cloud = "aws" + disk_size = 30 + events = (known after apply) + id = (known after apply) + image = "ubuntu" + logs = (known after apply) + machine = "t3.small" + name = "tpi-examples-basic " + parallelism = 1 + region = "us-west" + script = <<-EOT #!/bin/bash sudo apt update sudo apt install -y python3-pip pip3 install -r requirements.txt python3 src/train.py EOT + spot = 0 + ssh_private_key = (sensitive value) + ssh_public_key = (sensitive value) + status = (known after apply) + timeout = 86400 + storage { + output = "shared" + workdir = "." } } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes 2022-04-26T16:28:30.881-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:28:30.912-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:28:30.912-0700 2022-04-26T16:28:30.935-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:28:30.935-0700 iterative_task.tpi-examples-basic: Creating... 2022-04-26T16:28:30.939-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700 2022-04-26T16:28:30.939-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700 2022-04-26T16:28:30.939-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700 2022-04-26T16:28:30.939-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:28:30 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:28:30.939-0700 TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait. ╷ │ Error: operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: 59A2YHE2FD93V5VE, HostID: ZhiCW18MkagKK0DmFj2pSZvrkLAANKr3KtFG28hidkuqXbSbe4lO0OUeBhNjxEae5Oh3zojGPts=, api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again. │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ ╷ │ Error: failed to create │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ ```

`TF_LOG_PROVIDER=TRACE terraform apply`

``` dabarnes@ubuntu:~/tests/blog-tpi-bees$ TF_LOG_PROVIDER=TRACE terraform apply 2022-04-26T16:35:20.724-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:35:20.731-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9] 2022-04-26T16:35:20.732-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55570 2022-04-26T16:35:20.732-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 2022-04-26T16:35:20.756-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.756-0700 2022-04-26T16:35:20.764-0700 [DEBUG] provider: using plugin: version=5 2022-04-26T16:35:20.764-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin181832840 network=unix timestamp=2022-04-26T16:35:20.763-0700 2022-04-26T16:35:20.775-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:20.777-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2022-04-26T16:35:20.779-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55570 2022-04-26T16:35:20.779-0700 [DEBUG] provider: plugin exited 2022-04-26T16:35:20.780-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:35:20.787-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9] 2022-04-26T16:35:20.788-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55578 2022-04-26T16:35:20.788-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 2022-04-26T16:35:20.812-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.811-0700 2022-04-26T16:35:20.820-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin2892143703 network=unix timestamp=2022-04-26T16:35:20.819-0700 2022-04-26T16:35:20.820-0700 [DEBUG] provider: using plugin: version=5 2022-04-26T16:35:20.831-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:20.835-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:20 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:20.835-0700 2022-04-26T16:35:20.836-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2022-04-26T16:35:20.838-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55578 2022-04-26T16:35:20.838-0700 [DEBUG] provider: plugin exited 2022-04-26T16:35:20.838-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:35:20.846-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9] 2022-04-26T16:35:20.846-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55587 2022-04-26T16:35:20.846-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 2022-04-26T16:35:20.870-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:20.870-0700 2022-04-26T16:35:20.878-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin3554886002 network=unix timestamp=2022-04-26T16:35:20.878-0700 2022-04-26T16:35:20.878-0700 [DEBUG] provider: using plugin: version=5 2022-04-26T16:35:20.890-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:20.893-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:20 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:20.893-0700 2022-04-26T16:35:20.895-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2022-04-26T16:35:20.897-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55587 2022-04-26T16:35:20.897-0700 [DEBUG] provider: plugin exited Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # iterative_task.tpi-examples-basic will be created + resource "iterative_task" "tpi-examples-basic" { + addresses = (known after apply) + cloud = "aws" + disk_size = 30 + events = (known after apply) + id = (known after apply) + image = "ubuntu" + logs = (known after apply) + machine = "t3.small" + name = "tpi-examples-basic " + parallelism = 1 + region = "us-west" + script = <<-EOT #!/bin/bash sudo apt update sudo apt install -y python3-pip pip3 install -r requirements.txt python3 src/train.py EOT + spot = 0 + ssh_private_key = (sensitive value) + ssh_public_key = (sensitive value) + status = (known after apply) + timeout = 86400 + storage { + output = "shared" + workdir = "." } } Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes 2022-04-26T16:35:24.512-0700 [INFO] provider: configuring client automatic mTLS 2022-04-26T16:35:24.520-0700 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 args=[.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9] 2022-04-26T16:35:24.520-0700 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55612 2022-04-26T16:35:24.520-0700 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 2022-04-26T16:35:24.546-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: configuring server automatic mTLS: timestamp=2022-04-26T16:35:24.546-0700 2022-04-26T16:35:24.554-0700 [DEBUG] provider: using plugin: version=5 2022-04-26T16:35:24.554-0700 [DEBUG] provider.terraform-provider-iterative_v0.10.9: plugin address: address=/tmp/plugin1681772421 network=unix timestamp=2022-04-26T16:35:24.553-0700 2022-04-26T16:35:24.567-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:24.570-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2022-04-26T16:35:24.570-0700 iterative_task.tpi-examples-basic: Creating... 2022-04-26T16:35:24.572-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "events" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700 2022-04-26T16:35:24.572-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "logs" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700 2022-04-26T16:35:24.572-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "addresses" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700 2022-04-26T16:35:24.572-0700 [INFO] provider.terraform-provider-iterative_v0.10.9: 2022/04/26 16:35:24 [DEBUG] setting computed for "status" from ComputedKeys: timestamp=2022-04-26T16:35:24.572-0700 TPI [INFO] Creation may take several minutes (consider increasing `timeout` https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#timeout). Please wait. 2022-04-26T16:35:24.574-0700 [TRACE] provider.stdio: received data: channel=STDERR len=110 2022-04-26T16:35:24.574-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:24.857-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:24.857-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:24.936-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:24.936-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:25.481-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:25.481-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:25.988-0700 [TRACE] provider.stdio: received data: channel=STDERR len=220 2022-04-26T16:35:25.988-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.070-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.070-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.153-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.153-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.422-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.422-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.458-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.458-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.823-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.823-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:26.887-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:26.887-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:27.185-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:27.186-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:27.265-0700 [TRACE] provider.stdio: received data: channel=STDERR len=110 2022-04-26T16:35:27.265-0700 [TRACE] provider.stdio: waiting for stdio data 2022-04-26T16:35:27.305-0700 [TRACE] provider.stdio: received data: channel=STDERR len=55 2022-04-26T16:35:27.305-0700 [TRACE] provider.stdio: waiting for stdio data ╷ │ Error: operation error S3: CreateBucket, https response error StatusCode: 409, RequestID: AHVV3JBFBP0KJNW2, HostID: aQNQT3SprqbnqOnO6BqftOJazOqhXLb2srjIDM7eZeiOR9zMQl8hBvws4yzZEhJ8WfOv9n3rtyw=, api error OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again. │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ ╷ │ Error: failed to create │ │ with iterative_task.tpi-examples-basic, │ on main.tf line 5, in resource "iterative_task" "tpi-examples-basic": │ 5: resource "iterative_task" "tpi-examples-basic" { │ ╵ 2022-04-26T16:35:27.309-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing" 2022-04-26T16:35:27.312-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/iterative/iterative/0.10.9/linux_amd64/terraform-provider-iterative_v0.10.9 pid=55612 2022-04-26T16:35:27.312-0700 [DEBUG] provider: plugin exited ```

0x2b3bfa0 commented 2 years ago

_name - (Optional) Discouraged and may be removed in future - change the resource name instead, i.e. resource "iterative_task" "some_other_example_name". Deterministic task name (e.g. name="Hello, World!" always produces id="tpi-hello-world-5kz6ldls-57wo7rsp")._

0x2b3bfa0 commented 2 years ago

That's the “expected” behavior and there is nothing we can do to mitigate it. You can avoid it by not specifying the name field, so a new name is generated for every run.

dacbd commented 2 years ago

hmmm, but why the name a conflict for the create bucket operation?

AWS event:

``` "eventTime": "2022-04-27T00:21:03Z", "eventSource": "s3.amazonaws.com", "eventName": "CreateBucket", "awsRegion": "us-west-1", "sourceIPAddress": "oops", "userAgent": "[aws-sdk-go-v2/1.11.0 os/linux lang/go/1.17.8 md/GOOS/linux md/GOARCH/amd64 api/s3/1.18.0]", "errorCode": "OperationAborted", "errorMessage": "A conflicting conditional operation is currently in progress against this resource. Please try again.", "requestParameters": { "CreateBucketConfiguration": { "LocationConstraint": "us-west-1", "xmlns": "http://s3.amazonaws.com/doc/2006-03-01/" }, "bucketName": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry", "Host": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry.s3.us-west-1.amazonaws.com" }, "responseElements": null, "additionalEventData": { "SignatureVersion": "SigV4", "CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256", "bytesTransferredIn": 153, "AuthenticationMethod": "AuthHeader", "x-amz-id-2": "d5ltZLOpYM0MCnDXf0cj/188ucBmIJW/IwMgeSUOyGlwHpt45y7hlTVEgWLIWlz1HIoD292Ztwc=", "bytesTransferredOut": 335 }, "requestID": "89DCAM5G4V7HMDB9", "eventID": "e62864f9-84dd-4df8-afaa-4cd7e92f19f2", "readOnly": false, "eventType": "AwsApiCall", "managementEvent": true, "recipientAccountId": "548501963152", "eventCategory": "Management", "tlsDetails": { "tlsVersion": "TLSv1.2", "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256", "clientProvidedHostHeader": "tpi-tpi-examples-basic-63n5cpxk-2tqun9ry.s3.us-west-1.amazonaws.com" } } ```

0x2b3bfa0 commented 2 years ago

why the name a conflict for the create bucket operation?

Because the old & new buckets had exactly the same name.

0x2b3bfa0 commented 2 years ago

You were trying to create a new bucket while the old one was still queued for deletion, and both had the same name.

dacbd commented 2 years ago

I see, and that takes time on the AWS for them to fully complete, and thus the error.

iterative / terraform-provider-iterative

`task` Create Bucket fails 409 #541

Relevant terraform outputs: