EventHub `capture_description.destination` should now support Data Lake

andrewbolster commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.5.7

AzureRM Provider Version

3.72

Affected Resource(s)/Data Source(s)

azurerm_eventhub

Terraform Configuration Files

provider "azurerm" {
    features {}
}

provider "random" {}

resource "azurerm_resource_group" "example" {
  name     = "example-rg"
  location = "East US"
}

resource "random_id" "storage_account_name" {
  byte_length           = 8
}

resource "azurerm_storage_account" "example" {
  name                     = lower(random_id.storage_account_name.hex)
  resource_group_name      = azurerm_resource_group.example.name
  location                 = azurerm_resource_group.example.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  access_tier              = "Cool"
  is_hns_enabled           = false
  min_tls_version      = "TLS1_2"
  public_network_access_enabled = false
  blob_properties {
    delete_retention_policy {
      days    = 7
    }
    container_delete_retention_policy {
      days    = 7
    }
  }
}

### FADA Data Lake
resource "azurerm_storage_data_lake_gen2_filesystem" "example" {
  name                  = "example"
  storage_account_id= azurerm_storage_account.example.id
}

resource "azurerm_eventhub_namespace" "example" {
    name                = "example-namespace"
    location            = azurerm_resource_group.example.location
    resource_group_name = azurerm_resource_group.example.name
    sku                 = "Standard"
}

resource "azurerm_eventhub" "example" {
    name                = "example-eventhub"
    namespace_name      = azurerm_eventhub_namespace.example.name
    resource_group_name = azurerm_eventhub_namespace.example.resource_group_name
    partition_count     = 1
    message_retention   = 1
    capture_description {
      enabled             = true
      encoding = "Avro"
      interval_in_seconds = 300
      size_limit_in_bytes = 10485760
      skip_empty_archives = true
      destination {
        name = "EventHubArchive.AzureDataLake"
        storage_account_id = azurerm_storage_account.example.id
        blob_container_name = "dev"
        archive_name_format = "{Namespace}/year={Year}/month={Month}/day={Day}/hour={Hour}/minute={Minute}/second={Second}/topic={EventHub}/partition={PartitionId}/"
      }
    }
}

Debug Output/Panic Output

│ Error: expected capture_description.0.destination.0.name to be one of ["EventHubArchive.AzureBlockBlob"], got EventHubArchive.AzureDataLake
│
│   with azurerm_eventhub.example,
│   on test.tf line 62, in resource "azurerm_eventhub" "example":
│   62:         name = "EventHubArchive.AzureDataLake"
│

Expected Behaviour

Successfully deploying an event hub capture destination to Azure Data Lake

Actual Behaviour

Rejected after plan.

Steps to Reproduce

terraform init terraform plan #successful terraform apply #failure

Important Factoids

No response

References

Upstream (now resolved) API issue: https://github.com/Azure/azure-rest-api-specs/issues/2255
Previous Provider Issue/Thread: https://github.com/hashicorp/terraform-provider-azurerm/pull/17220

rcskosir commented 1 year ago

@xiaxyi, could you take a look at redoing your previous PR (#17220) using gen2?

enikkari commented 9 months ago

Should this be working now? I get the same error

expected capture_description.0.destination.0.name to be one of ["EventHubArchive.AzureBlockBlob"], got EventHubArchive.AzureDataLake

I use: Terraform v1.7.3 on darwin_arm64

provider registry.terraform.io/hashicorp/azurerm v3.91.0

It is not quite clear from all the git issues and documentation whether the gen2 datalake capture is supported or not.

repcaks commented 8 months ago

Hi, any updates? I see that eventhub in azurerm provider is quite outdated in other features

hashicorp / terraform-provider-azurerm