Destroy Turbot resources doesn't validate child iaas resources

[ambrosy-eric]

Issue

Some turbot policy setting resources, such as gcp event handler enforcement, create resources in a targeted cloud account. When the turbot_policy_setting resource is destroyed, the underlying cloud resource created by the policy is not destroyed and therefore when a new policy is created, it still passes even though the child resource(s) are not successfully created and the console reports an error.

Example

The below creates a pubsub topic and subscription in a specified gcp account.

resource "turbot_policy_setting" "pubsub_enable" {
  resource = turbot_resource.gcp_project.id
  type     = "tmod:@turbot/gcp-pubsub#/policy/types/pubsubApiEnabled"
  value    = "Enforce: Enabled"
}

resource "turbot_policy_setting" "event_handler_pubsub" {
  resource = turbot_resource.gcp_project.id
  type     = "tmod:@turbot/gcp#/policy/types/eventHandlersPubSub"
  value    = "Enforce: Configured"
}

When you delete these resources, the gcp resources remain deployed.

Additionally when you create a new pubsub policy for event handlers, it will successfully apply but does not modify the existing subscription or add a new one. So is not actually applied successfully as the pubsub is not able to successfully send to turbot. The turbot console does report this error as Terraform error executing stack but the apply still succeeds with not indication that there was an issue

Expectation

• Turbot resources that create client resources in a project/account are able to clean up after themselves
• Terraform apply fails when a resource create fails

Terraform version

$ terraform --version
Terraform v0.12.28
+ provider.turbot v1.3.0

[kaidaguerre]

We are looking into this.

[kaidaguerre]

Thanks for raising this issue. We appreciate the time taken to document it.

I have reproduced the problem and I'm investigating the cause and potential fixes.

[kaidaguerre]

Here is an update on this issue:

There are 2 separate aspects to this.

Behaviour when the policy setting resources were deleted

Policy settings were created using the Turbot Terraform provider. These settings enabled the GCP pub-sub stack which then created the required resources.

The policy settings were then deleted. When the eventHandlersPubSub policy setting was deleted, the value for that policy was derived from the default policy value, which is skip. Therefore the pubsub stack control went into a skipped skipped state, leaving existing resources in place.

This is expected behaviour

Behaviour when the policy settings are recreated

The problem was that when the policy value was re-added, rather than recognising the existing resources, the pubsub stack tried to create new ones, resulting in errors.

This is an issue with the Turbot Enterprise software, rather than a provider issue, and we are actively pursuing the problem.

I will continue to track it here until it is resolved.

[kaidaguerre]

A further update on this one.

We believe the problem was caused by an interesting timing issue in the stack code.

The policy GCP > Turbot > Event Handlers > Pub/Sub policy controls the running of the the GCP > Turbot > Event Handlers> Pub/Sub stack. The Terraform source for this stack is provided by the calculated policy GCP > Turbot > Event Handlers > Pub/Sub > Source

Here is the flow of events for the scenario:

Policy Setting Creation When the GCP > Turbot > Event Handlers > Pub/Sub policy setting of Enforce: Configured is created, the following happens:

1. the policyGCP > Turbot > Event Handlers > Pub/Sub > Source is calculated - this will be a Terraform configuration containing all required resources
2. the stack control will run to create these resources

Policy Setting Deletion When the setting for GCP > Turbot > Event Handlers > Pub/Sub setting is deleted, the default value for the policy, Skip, is used. This causes 2 things to happen: 1) the stack control goes into a skipped state 2) the policyGCP > Turbot > Event Handlers > Pub/Sub > Source is calculated to be an empty Terraform configuration

Policy Setting Re-Creation When GCP> Turbot> Event Handlers> Pub/Sub is set to Enforce: Configured once again, the following happens: 1) the stack control runs. As the source policy is still empty, the stack will delete all the resources it previously created 2) the policyGCP > Turbot > Event Handlers > Pub/Sub > Source is calculated to contain resources once again 3) The calculation of the Source policy triggers the stack to run again and create the resources once again.

The rapid deletion and re-creation of the same resources triggered a race condition with the gcp deletion/creation events which caused the stack error reported.

The fix we will be releasing is an update to the GCP PubSub mod to NOT set the Source policy to empty when the GCP > Turbot > Event Handlers > Pub/Sub is set to Skip. This will avoid the resources being unnecessarily deleted when the stack is re-enabled.