Support for Azure Event Hub - Log compaction Feature

ValentinVers commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When creating an Azure Event Hub cleanup policy can be set to "Delete" with a particular retention time or to "Compact" with a Tombstone retention time.

This new feature is currently not supported in azurerm_eventhub.

New or Affected Resource(s)/Data Source(s)

azurerm_eventhub

Potential Terraform Configuration

resource "azurerm_eventhub" "example" {
  name                = "acceptanceTestEventHub"
  namespace_name      = azurerm_eventhub_namespace.example.name
  resource_group_name = azurerm_resource_group.example.name
  partition_count     = 2
  cleanup_policy      = "Compact" // or "Delete"
  tombstone_retention = 1 // In hours
}

References

repcaks commented 6 months ago

Any updates or planned delivery date?

dico-harigkev commented 4 months ago

This feature is still missing and would be important for a use case we have.

Also, trying to import an Azure EventHub instance (azurerm_eventhub) which has been created with a cleanup policy of "Compact" through the Azure Portal even crashes the azurerm provider, because in that case the API seems to return a value for message_retention that can not be parsed as integer (too large, > 2^63):

╷
│ Error: Request cancelled
│ 
│   with azurerm_eventhub.example,
│   on example.tf line 31, in resource "azurerm_eventhub" "example":
│   31: resource "azurerm_eventhub" "example" {
│ 
│ The plugin.(*GRPCProvider).ReadResource request was cancelled.
╵

Stack trace from the terraform-provider-azurerm_v3.113.0_x5 plugin:

panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

goroutine 202 [running]:
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).get(0xc003202180, {0xc0032012e0, 0x1, 0x1}, 0x8?)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource_data.go:556 +0x333
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).State(0xc003202180)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource_data.go:354 +0x473
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ShimInstanceStateFromValue(0xc000cffc00, {{{0x8a58448?, 0xc000631790?}}, {0x7306ac0?, 0xc003206c60?}})
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/resource.go:670 +0x1d8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000413b90, {0x8a558c0?, 0xc003206360?}, 0xc0031dc840)
        github.com/hashicorp/terraform-plugin-sdk/v2@v2.29.0/helper/schema/grpc_provider.go:623 +0x1f3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc000aa7220, {0x8a558c0?, 0xc0031fbb90?}, 0xc003198b40)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/tf5server/server.go:789 +0x48b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x7ed90e0?, 0xc000aa7220}, {0x8a558c0, 0xc0031fbb90}, 0xc000ab5490, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000038000, {0x8a805c0, 0xc001c041a0}, 0xc0031fd440, 0xc001bd69c0, 0xe081468, 0x0)
        google.golang.org/grpc@v1.58.3/server.go:1374 +0xde7
google.golang.org/grpc.(*Server).handleStream(0xc000038000, {0x8a805c0, 0xc001c041a0}, 0xc0031fd440, 0x0)
        google.golang.org/grpc@v1.58.3/server.go:1751 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/grpc@v1.58.3/server.go:986 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 52
        google.golang.org/grpc@v1.58.3/server.go:997 +0x145

Error: The terraform-provider-azurerm_v3.113.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

rutgersc commented 4 months ago

Last time I've looked at this (btw I don't know what I'm doing), there was some mismatch betweeen the go-azure-sdk api version and the actual api versions deployed by azure : https://github.com/hashicorp/go-azure-sdk/issues/976. There has been some update, so maybe it's fixed now.

This is what I had been working with: https://github.com/hashicorp/terraform-provider-azurerm/compare/main...rutgersc:terraform-provider-azurerm:main2.

deichrenner commented 3 months ago

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

rutgersc commented 3 months ago

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

No, not yet. Running into the same issue as dico-harigkev, where it's unable to parse into an int panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

This now also applies to the new RetentionDescription.RetentionTimeInHours:

If cleanupPolicy is Compact the returned value of this property is Long.MaxValue.

Not sure what to do here. Of the available schema types, only int is currently available: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/tf/pluginsdk/schema.go. Maybe someone familiar with the codebase can weigh in on this? @tombuildsstuff?

This branch is able to create the eventhub with the correct cleanup policy.

rickardp commented 1 month ago

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

No, not yet. Running into the same issue as dico-harigkev, where it's unable to parse into an int panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

This now also applies to the new RetentionDescription.RetentionTimeInHours:

If cleanupPolicy is Compact the returned value of this property is Long.MaxValue.

Not sure what to do here. Of the available schema types, only int is currently available: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/tf/pluginsdk/schema.go. Maybe someone familiar with the codebase can weigh in on this? @tombuildsstuff?

This branch is able to create the eventhub with the correct cleanup policy.

What happened with this PR? Any input from maintainers?

rutgersc commented 1 month ago

What happened with this PR? Any input from maintainers?

No idea, there isn't a PR yet. I likely won't spend more time on this either, maybe someone else can pick this up.

hashicorp / terraform-provider-azurerm