Add Support for Export Cost Management data to secure storage accounts with firewall

robertbor commented 1 year ago

Is there an existing issue for this?

[x] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Description

At September 12, 2023 Microsoft announced that it is possible to send the cost export data to a storage account behind a firewall.

https://azure.microsoft.com/en-us/updates/exportbehindfirewallga/

With the current provider version: 3.81.0 it is not possible to configure it.

Error: creating Scoped Export (Scope: "/subscriptions/***" │ Export Name: "exportname"): unexpected status 400 with error: 400: The exports service is not authorized to access the specified storage account 'storageaccountname'. For more information see: https://go.microsoft.com/fwlink/?linkid=2127915 │ │ with azurerm_subscription_cost_management_export.cost_export["costexportname"], │ on cost-export.tf line 48, in resource "azurerm_subscription_cost_management_export" "cost_export": │ 48: resource "azurerm_subscription_cost_management_export" "cost_export" *** │ │ creating Scoped Export (Scope: │ "/subscriptions/***" │ Export Name: "exportname"): unexpected status 400 with error: 400: The exports │ service is not authorized to access the specified storage account

New or Affected Resource(s)/Data Source(s)

azurerm_subscription_cost_management_export

Potential Terraform Configuration

No response

References

No response

rcskosir commented 1 year ago

Thank you for taking the time to open this feature request!

bytef10w commented 9 months ago

Any updates on this? We faced exactly the same issue. It works fine while creating manually. Tested with 3.84.0 and latest one

Tomasz-Kluczkowski commented 7 months ago

any progress? got hit by this too....

philippfrenzel commented 3 months ago

As I wanna get updates to this issue - i put myself in as commenter- this issue is blocking us - so would be awesome if someone has a solution or workaround

emad0082 commented 2 months ago

I am experiencing similar issues on this. Any updates?

w0ut0 commented 2 months ago

In my experience, this only works only by using Microsoft.CostManagement/exports api version 2023-07-01-preview, not with the newer version 2023-11-01 (which complains that the storage account is not accessible, even if all prerequisites are fine).

emad0082 commented 1 month ago

@w0ut0

Do you have any examples of how to do this?

We are attempting to create the resource programmatically

w0ut0 commented 1 month ago

Well, I had an example like

resource "time_static" "start_time" {}

# https://github.com/hashicorp/terraform-provider-azurerm/issues/23962
resource "azapi_resource" "cost_exports" {
  lifecycle {
    ignore_changes = [body.properties.schedule.recurrencePeriod.from]
  }
  type      = "Microsoft.CostManagement/exports@2023-07-01-preview"
  name      = "export"
  parent_id = data.azurerm_subscription.current.id
  location  = "global"
  identity {
    type = "SystemAssigned"
  }

  body = {
    properties = {
      definition = {
        dataSet = {
          configuration = {
            dataVersion = "2021-10-01",
            filters     = []
          }
          granularity = "Daily"
        }
        timeframe = "MonthToDate"
        type      = "AmortizedCost"
      }
      deliveryInfo = {
        destination = {
          container      = "azure"
          rootFolderPath = "cost",
          type           = "AzureBlob",
          resourceId     = "<storage_account_id>"
        }
      }
      format                = "Csv"
      partitionData         = true
      dataOverwriteBehavior = "OverwritePreviousReport"
      compressionMode       = "gzip"
      schedule = {
        recurrence = "Daily"
        status     = "Active"
        recurrencePeriod = {
          from = timeadd(time_static.start_time.id, "1h")
          to   = "2029-07-01T00:00:00Z"
        }
      }
    }
  }
}

But it does not apply in our CI/CD because of https://github.com/Azure/terraform-provider-azapi/issues/575, which I hope is fixed with https://github.com/Azure/terraform-provider-azapi/pull/616 (but not yet released)

nikolai-fra commented 1 month ago

In the code client.go here is defined, that the underlaying azure sdk is using API version "2021-10-01" for the exports. This has to be changed to "2023-11-01" (see here)

Does somebody here knows how to test this provider for implications resulting from the change in the api version?

chirangaalwis commented 1 week ago

In the code client.go here is defined, that the underlaying azure sdk is using API version "2021-10-01" for the exports. This has to be changed to "2023-11-01" (see here)

Does somebody here knows how to test this provider for implications resulting from the change in the api version?

Folks, any update on this? This has so far forced us to create the used Storage Account without restricted network access. Appreciate it if this suggested fix is introduced asap.

emad0082 commented 1 week ago

@chirangaalwis use the REST API or the solution provided by w0ut0

hashicorp / terraform-provider-azurerm