Support for Authentication Identity when Capture for Event Hub is enabled

andrilareida commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

In the Azure Portal in the Capture Menu of an Event Hub, I can choose among Authentication Options: SAS, System Assigned Identity, and User Assigned Identity:

I would like to have this option in Terraform when defining a Capture Block with Destination in an azurerm_eventhub resource.

New or Affected Resource(s)/Data Source(s)

azurerm_eventhub

Potential Terraform Configuration

resource "azurerm_eventhub" "example" {
  name                = "acceptanceTestEventHub"
  namespace_name      = azurerm_eventhub_namespace.example.name
  resource_group_name = azurerm_resource_group.example.name
  partition_count     = 2
  message_retention   = 1
  capture_description {
    enabled  = true
    encoding = "Avro"
    destination {
      name                = "EventHubArchive.AzureBlockBlob"
      archive_name_format = "{Namespace}/{EventHub}/{Year}/{Month}/{Day}/{PartitionId}/..."
      blob_container_name = azurerm_storage_data_lake_gen2_filesystem.branch.name
      storage_account_id  = azurerm_storage_account.capture.id
      //System Assigned
      identity {
        type = "SystemAssigned"
      }
      //User Assigned
      identity {
        type        = "UserAssigned"
        identity_id = "object-id"
      }
    }
  }
}

References

No response

writetoarun commented 8 months ago

does any one know how we can do this in Azure CLI? or any other work arounds meanwhile ?

miguelillo commented 5 months ago

Any updates on this?

ottensjors commented 4 weeks ago

managed identity inheritence of eventhub namespace with terraform

we have a usecase where we have een eventhub namespace with an eventhub that needs to capture the events to an azure blog storage container . we want to do this without providing SAS-tokens. so what we did is the following:

set up the 'azure eventhub namespace' with SystemAssigned identity
provide the SystemAssigned identity with storage blob data contributor on the storage account / storage container
create an ' azure eventhub'. within the 'azure_eventhub_namespace' created in [1] that receives events and captures them to the blob storage container

by doing so it seems that the azure_eventhub' inherits the permisisons of the 'azure_eventhub_namespace". and is able to write the received events to the blob storage container configured in the capture_destinaion block of the code below

code:

#### AZURE EVENTHUB NAMESPACE
resource "azurerm_eventhub_namespace" "ffa_titan_evenhub_namespace" {
  name                = var.eventhub_namespace
  location            = var.location
  resource_group_name = var.resourcegroup_name
  sku                 = "Standard"
  capacity            = 2
  public_network_access_enabled = true #for testing purposes we enabled public access as its an endpoint that needs to be able to receive data (need to find proper networking solution)

  identity {
    type = "SystemAssigned"
  }
}

# Provide EVETHUB NAMESPACE MSI the role of 'Storage Blob Data Contributor' in Storage Account ADLS
resource "azurerm_role_assignment" "ffa_titan_eventhub_namespace_dfs_role_assignment" {
  scope                = azurerm_storage_account.ffa_titan_adls_gen2.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_eventhub_namespace.ffa_titan_evenhub_namespace.identity[0].principal_id
}

### AZURE EVENTHUB
resource "azurerm_eventhub" "ffa_titan_eventhub" {
  name                = "acceptanceTestEventHub"
  namespace_name      = azurerm_eventhub_namespace.ffa_titan_evenhub_namespace.name
  resource_group_name = var.resourcegroup_name
  partition_count     = 2
  message_retention   = 1
  depends_on = [ 
    azurerm_eventhub_namespace.ffa_titan_evenhub_namespace
   ]
   capture_description {
    enabled = true
    encoding = "Avro"
    destination {
      name = "EventHubArchive.AzureBlockBlob"
      archive_name_format = "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}"
      blob_container_name = "${var.storage_account}-bronze"
      storage_account_id = azurerm_storage_account.ffa_titan_adls_gen2.id
    }
   }
}

by doing so the 'azure_eventhub'. was able to sink data to our azure storage container without the need for any SAS token and we did not have to put in a workaround for setting the 'Authentication for Capture' property.

the result is as follows when sending test-data to it:

Powershell eventhub IdentityType-property workaround

Although I do find that it should be supported in the 'azurerm_eventhub' module within terraform.

the workaround would be to use an additional powershellscript and run it as a null_resource in terraform to set the 'Authentication for Capture' property to System Assigned identity with this powershellscript:

# set vars
$azureResourceGroupName = "<your-resourcegroup-name>"
$eventHubNameSpace = "<your-evethub-namespace>"
$eventHubName = "<your-eventhub-name>"

#W get PRE-MANAGED-IDENTITY eventhub details 
Write-Output "Getting PRE-MANAGED-IDENTITY eventhub  details"
$eventhub = Get-AzEventHub -ResourceGroupName $azureResourceGroupName -NamespaceName $eventHubNameSpace -Name $eventHubName
Write-Output $eventhub
Write-Output "=> Finished Getting PRE-MANAGED-IDENTITY eventhub  details"

## Set IdentityType property tot 'SystemAssigned' on eventhub
Write-Output "Setting Eventhub-property 'IdentityType'  to 'SystemAssigned'"
Set-AzEventHub -ResourceGroupName $azureResourceGroupName -NamespaceName $eventHubNameSpace -Name $eventHubName -IdentityType "SystemAssigned"
Write-Output "=> FInished Setting Eventhub-property 'IdentityType'  to 'SystemAssigned'"

## wait for change to be processed in azure eventhub
Write-Output "Waiting for IdentityType to be set"
Start-Sleep -s 5
Write-Output "=> Finished Waiting for IdentityType to be set"

# Get POST-MANAGED-IDENTITY eventhub details
Write-Output "Getting POST-MANAGED-IDENTITY eventhub  details"
$eventhubPost = Get-AzEventHub -ResourceGroupName $azureResourceGroupName -NamespaceName $eventHubNameSpace -Name $eventHubName
Write-Output $eventhubPost
Write-Output "=> Finished Getting POST-MANAGED-IDENTITY eventhub  details"

result of script:

dawsonar802 commented 2 weeks ago

I ran into the same issues and worked through deploying the Eventhub through the azurerm resource and doing an update on the resource via the AzAPI provider. The key thing to note here is make sure to use the correct API version, which needs to be 2023-01-01-preview or maybe higher as the older versions do not seem to have identity as an option.

Ignore on the capture_description is needed as well due to Terraform trying to remove the capture each time.

resource "azurerm_eventhub" "mod_eh" {
  name                = var.name
  namespace_name      = var.namespace_name
  resource_group_name = var.resource_group_name

  partition_count   = var.partition_count
  message_retention = var.message_retention

  # dynamic "capture_description" {
  #   for_each = var.capture.enabled == true ? [1] : []
  #   content {
  #     enabled             = var.capture.enabled
  #     encoding            = lookup(var.capture, "encoding", "Avro")
  #     interval_in_seconds = lookup(var.capture, "interval_in_seconds", 300)
  #     size_limit_in_bytes = lookup(var.capture, "size_limit_in_bytes", 314572800) # 300 MB
  #     skip_empty_archives = lookup(var.capture, "skip_empty_archives", false)
  #     destination {
  #       name                = var.capture.destination_name
  #       storage_account_id  = var.capture.storage_account_id
  #       blob_container_name = var.capture.blob_container_name
  #       archive_name_format = var.capture.archive_name_format
  #     }
  #   }
  # }
  lifecycle {
    ignore_changes = [capture_description]
  }
}

resource "azapi_update_resource" "eh_capture" {
  count = var.capture.enabled == true ? 1 : 0

  type      = "Microsoft.EventHub/namespaces/eventhubs@2023-01-01-preview"
  name      = azurerm_eventhub.mod_eh.name
  parent_id = var.namespace_id

  body = jsonencode({
    properties = {
      captureDescription = {
        enabled             = "${var.capture.enabled}"
        encoding            = lookup(var.capture, "encoding", "Avro")
        interval_in_seconds = lookup(var.capture, "interval_in_seconds", 300)
        size_limit_in_bytes = lookup(var.capture, "size_limit_in_bytes", 314572800) # 300 MB
        skip_empty_archives = lookup(var.capture, "skip_empty_archives", false)
        destination = {
          name = "EventHubArchive.AzureBlockBlob"
          properties = {
            storageAccountResourceId = "${var.capture.storage_account_id}"
            blobContainer            = "${var.capture.blob_container_name}"
            archiveNameFormat        = "${var.capture.archive_name_format}"
          }
          identity = {
            type = "SystemAssigned"
          }
        }
      }
    }
  })

  depends_on = [azurerm_eventhub.mod_eh]
}

hashicorp / terraform-provider-azurerm