Terraform Aqua provider doesn't properly destroy "aquasec_enforcer_groups"

[ruzickap]

Terraform Aqua provider doesn't properly destroy "aquasec_enforcer_groups".

Terraform Version

❯ terraform -v
Terraform v1.2.2
on darwin_amd64
+ provider registry.terraform.io/aquasecurity/aquasec v0.8.11

Affected Resource(s)

• aquasec_enforcer_groups

Steps to Reproduce

I'm using this code to create enforcer groups in Aqua:

terraform {
  backend "local" {
    path = "terraform.tfstate"
  }
  required_providers {
    aquasec = {
      version = "0.8.11"
      source  = "aquasecurity/aquasec"
    }
  }
}

provider "aquasec" {
  username = var.aquasec_username
  aqua_url = var.aquasec_url
  password = var.aquasec_password
}

variable "aquasec_username" {
  type        = string
  description = "Aqua Username"
}

variable "aquasec_password" {
  type        = string
  description = "Aqua Password"
}

variable "aquasec_url" {
  type        = string
  description = "Aqua URL"
}

resource "aquasec_enforcer_groups" "auqa-enforcers-agent-kubernetes" {
  group_id    = "tes-enforcer"
  type        = "agent"
  description = "Test Enforcer"
  enforce     = true
  # Host Assurance
  host_assurance = true
  # Network Firewall (Host Protection)
  host_network_protection = true
  # Runtime Controls
  host_protection = true
  # Network Firewall (Container Protection)
  network_protection = true
  # Advanced Malware Protection (Container Protection)
  container_antivirus_protection = true
  # Runtime Controls
  container_activity_protection = true
  # Image Assurance
  image_assurance = true
  # Advanced Malware Protection (Host Protection)
  antivirus_protection = true
  # Host Images
  sync_host_images = true
  # Risk Explorer
  risk_explorer_auto_discovery = true

  orchestrator {
    type      = "kubernetes"
    namespace = "aqua"
  }
}

resource "aquasec_enforcer_groups" "auqa-enforcers-kube_enforcers-kubernetes" {
  group_id    = "test-kube_enforcer"
  type        = "kube_enforcer"
  description = "Test Kube Enforcer"
  enforce     = true
  # Enable admission control
  admission_control = true
  # Perform admission control if not connected to a gateway
  block_admission_control = true
  # Enable workload discovery
  auto_discovery_enabled = true
  # Register discovered pod images
  auto_scan_discovered_images_running_containers = true
  # Add discovered registries
  auto_discover_configure_registries = true
  # Kube-bench image path
  kube_bench_image_name = "registry.aquasec.com/kube-bench:v0.6.5"
  # Secret that holds the registry credentials for the Pod Enforcer and kube-bench
  micro_enforcer_secrets_name = "aqua-registry"
  # Auto copy these secrets to the Pod Enforcer namespace and container
  auto_copy_secrets = true

  orchestrator {
    type      = "kubernetes"
    namespace = "aqua"
  }
}

Once the enforcer groups are create I connected my k8s cluster and everything is working like expected:

The problem appeared when I try to run terraform destroy where I'm getting:

aquasec_enforcer_groups.auqa-enforcers-kube_enforcers-kubernetes: Destroying... [id=test-kube_enforcer]
aquasec_enforcer_groups.auqa-enforcers-agent-kubernetes: Destroying... [id=tes-enforcer]
╷
│ Error: failed deleting enforcer group, status code: 400
│
│
╵
╷
│ Error: failed deleting enforcer group, status code: 400

When I delete the k8s cluster I can not destroy it either:

I need to manually delete connections in Aqua GUI and then I can destroy it:

-> Terraform should be able to destroy enforcer groups without any manual intervention.

---

Use case - we are creating / deleting k8s clusters for testing purposes therefore we would like to automate the whole process:

• create cluster
• register cluster to Aqua
• run tests
• delete cluster
• delete cluster form Aqua

[KoppulaRajender]

Hi @ruzickap - I am not able to reproduce your issue. I can successfully destroy the enforcer groups easily, without hassle.

aquasec_enforcer_groups.auqa-enforcers-agent-kubernetes: Destroying... [id=tes-enforcer]
aquasec_enforcer_groups.auqa-enforcers-kube_enforcers-kubernetes: Destroying... [id=test-kube_enforcer]
aquasec_enforcer_groups.auqa-enforcers-kube_enforcers-kubernetes: Destruction complete after 1s
aquasec_enforcer_groups.auqa-enforcers-agent-kubernetes: Destruction complete after 1s

Destroy complete! Resources: 2 destroyed.

I have used your provided resource definition for enforcer groups creation and terraform v1.2.2.

Can you please provide the aqua version and type (SaaS or Enterprise) causing this issue?

[ruzickap]

I'm using Aqua SaaS 2022.4.71. When you reproduce it please try all these steps:

• Create enforcer groups using the code above
• Register k8s cluster to Aqua sing newly provisioned enforcer group (enforcer + kube_enforcer) [agent + kube_enforcer]
  • ^^^ this is the important part to see the enforcer group and the connections like it is above on the first screenshot
• Destroy enforcer group using terraform

If you can not reproduce it - let me know - I'll record video showing the issue... The important part is to have k8s cluster connected to the enforcer groups created by terraform before you do terraform destroy.

[KoppulaRajender]

Hi @ruzickap this is a known issue in the product API and we have a fix for that. We have added a fix in #138 that supports deleting disconnected enforcers in the delete enforcer API. Currently, the relate_deleted feature waiting for the product team to add into the 2022.4 version but it is there in the 6.5 and lower versions. Will merge the PR once the feature adds to 2022.4 aqua.

Thanks,

[ruzickap]

This issue was not fixed in latest aquasec provider v0.8.12 - I created new story to investigate: https://github.com/aquasecurity/terraform-provider-aquasec/issues/156