azapi_resource_list and response_export_values support for more complex JSON

[DevopsMercenary]

I'm trying to get the name of all the file shares in a storage account. I run azapi_resource_list and returns the following JSON

  "value":
    [
      {
        "id": "/subscriptions/0000000-0000-0000-0000-000000000000/resourceGroups/maintenance-configuration/providers/Microsoft.Storage/storageAccounts/winfunapp/fileServices/default/shares/maintenance-function-app-bfc8",
        "name": "maintenance-function-app-bfc8",
        "type": "Microsoft.Storage/storageAccounts/fileServices/shares",
        "etag": "0x8DC8E72C1214326",
        "properties":
          {
            "leaseStatus": "unlocked",
            "leaseState": "available",
            "accessTier": "TransactionOptimized",
            "lastModifiedTime": "2024-06-17T02:11:09.0000000Z",
            "shareQuota": 102400,
            "enabledProtocols": "SMB"
          }
      }
    ]
}

Great, awesome. Now I see this response_export_values option in azapi_resource_list and I would like to just get the name to return...

And for the life of me I have no idea as to how I can do it. I've tried a bunch of options but doesn't seem to be possible for instance...

  response_export_values = ["value"]
  response_export_values = ["value.name"]
  response_export_values = ["value[].name"]
  response_export_values = ["value[0].name"]

[stemaMSFT]

Thanks for the issue @DevopsMercenary! This one is a lack of clarity that we may want to update in docs. Value is already implicitly included, so you’d want to do [“name”] I believe. Let me know if that doesn’t work. @ms-henglu for FYI

[ms-henglu]

Hi @DevopsMercenary ,

Thank you for taking time to report this issue!

The output of azapi_resource_list data source could be in HCL format if specifying enable_hcl_output_for_data_source = true in the provider block. The default output is JSON format, but it's also possible to use jsondecode function to parse it to an HCL object.

After that, you could use the splat expression to get the name.

Here's an example:

provider "azapi" {
  enable_hcl_output_for_data_source = true
}

data "azapi_resource_list" "test" {
  type = "Microsoft.Automation/automationAccounts@2023-11-01"
  parent_id = "/subscriptions/{subscription id}"
  response_export_values = ["*"]
}

output "o1" {
  // value = ["foo", "bar"]
  value = data.azapi_resource_list.test.output.value[*].name
}

More details about the splat expression: https://developer.hashicorp.com/terraform/language/expressions/splat

[smokedlinq]

Is the HCL output required for this to work? jmsepath on the json would be handy here...

[smokedlinq]

@ms-henglu I did try doing this with something like this but it's not filtering the output ...

terraform {
  required_providers {
    azapi = {
      source  = "Azure/azapi"
      version = "~> 1.13"
    }
  }
}

provider "azapi" {
  enable_hcl_output_for_data_source = true
}

variable "scope" {
  type = string
}

data "azapi_resource_list" "role_definitions" {
  parent_id              = var.scope
  type                   = "Microsoft.Authorization/roleDefinitions@2022-04-01"
  response_export_values = ["id", "properties.roleName"]
}

output "value" {
  value = data.azapi_resource_list.role_definitions.output
}

[smokedlinq]

Best I can tell from the filtering, it doesn't support properties that are arrays to filter the child values as well ...

[ms-henglu]

Hi @smokedlinq ,

Yes, we'll consider to support JMSEPath in the next major release.

Currently, we need to make the output as an HCL object first(either using the enable_hcl_output_for_data_source or the jsondecode function). After that, you could use Terraform functions to filter the result, for example, using the for expression to filter the result: https://developer.hashicorp.com/terraform/language/expressions/for

[smokedlinq]

Yes but it still stores the full response in the state file. This causes excessively large state files and we actually hit a limit because of this. Our use case is each subscription needs its role definitions list for lookups so we load that. That response alone is 700kb. Multiplied across 150 subscriptions that single lookup is 100+mb. Combine this with other use cases such as the list of subscriptions or role assignments and suddenly 1+gb state files are a thing...

[smokedlinq]

FYI the hcl output is awesome but the dynamic schema is stored per item of a list in these kind of responses causing the large state data demands.

[ms-henglu]

Thanks for the details! I've created this issue to track the JSMEPath support: https://github.com/Azure/terraform-provider-azapi/issues/535

[DevopsMercenary]

Is the HCL output required for this to work? jmsepath on the json would be handy here...

Totally Agree @smokedlinq . For now I did convert JSON to HCL and a bunch of local vars to carefully parse out what I wanted. JMSEPath would be awesome, to (1) only get what I want directly, and (2) to reduce the size of the state file.

Looking forward to https://github.com/Azure/terraform-provider-azapi/issues/535