"external" provider returns an invalid json

[ghost]

This issue was originally opened by @Phydeauxman as hashicorp/terraform#17632. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.11.3
+ provider.azurerm v1.2.0
+ provider.external v1.0.0

Terraform Configuration Files

variable ilbase_resourceId {
  default = "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"
}

data "external" "aseVip" {
  program = ["az", "resource", "show", "--ids", "${var.ilbase_resourceId}"]
}

Debug Output

Crash Output

Expected Behavior

The JSON object would be returned to terraform

Actual Behavior

Terraform config generates error:

data.external.aseVip: data.external.aseVip: command "az" produced invalid JSON: json: cannot unmarshal object into Go value of type string

Steps to Reproduce

Additional Context

If after I provision the App Service Environment I run the command below:

az resource show --ids "/subscriptions/<my_subscription>/resourceGroups/my-rg/providers/Microsoft.Web/hostingEnvironments/my-ilbase/capacities/virtualip"

It returns the JSON object below:

{
      "additionalProperties": {
        "internalIpAddress": "10.10.1.11",
        "outboundIpAddresses": [
          "52.224.70.119"
        ],
        "serviceIpAddress": "52.224.70.119",
        "vipMappings": []
      },
      "id": null,
      "identity": null,
      "kind": null,
      "location": null,
      "managedBy": null,
      "name": null,
      "plan": null,
      "properties": null,
      "sku": null,
      "tags": null,
      "type": null
}

References

[apparentlymart]

Hi @Phydeauxman! Sorry this didn't work as expected.

At the moment this data source is limited to dealing only with simple string values, because the provider model requires each attribute to have a type and this one is typed as "map of strings". The additionalProperties property in your example is hitting this error because it's not convertible to a string.

It's currently necessary to flatten the result to use only a single level of keys and only string values. For the lists in your example, it'd be required to use some sort of delimiter and then use the split function to divide the single string into a list.

We do intend to make this more flexible in future but need to do some other work in Terraform Core first so that it's possible within provider schema to describe the result type of this attribute. (Or more accurately, to describe that its type is not known until it is read.) Since it seems we don't already have an issue open to represent this, I'm going to label this one to represent that feature. We need to get through some other work first so there won't be any immediate action here, but we will return to this once that underlying work is done and share more details.

[Phydeauxman]

@apparentlymart Thanks for the detailed explanation.

[marinsalinas]

Hello @apparentlymart , What if we add a raw_result attribute which represents the raw output string from command executed and then use jsondecode built-in function to process the JSON.

eg.

data "external" "policies" {
  program = ["sh", "-c", <<EOT
echo `{"type":"directory","name":"policies","files":["file1", "file2", "file3"]}
EOT
  ]
}

locals {
  files =  jsondecode(data.external.policies.result_raw)
}

output "policies" {
  value =local.files
}

[rquadling]

My use case is to get the current desired capacity of an autoscaling group so that I can use that in the replacement autoscaling group. I've seen a Cloudformation solution, but that required more understanding than I have at the time.

Initially, I had thought that just returning the JSON from awscli would have been enough, but unfortunately, the result for the data_source.external.result is a list of strings. No numbers, arrays, maps, lists.

So inspired by a comment made by Marin Salinas, I found that Terraform can access local files!

In addition, I am using an assumed role to do the work.

And so I documented my solution here.

[rquadling]

Just like to update the above with https://gist.github.com/rquadling/c2ee7f38ccac229673c1e7aabe1ad926 and https://registry.terraform.io/modules/digitickets/cli/aws/1.1.0

[jeremykatz]

Hello @apparentlymart , What if we add a raw_result attribute which represents the raw output string from command executed and then use jsondecode built-in function to process the JSON.

Raw output and raw input sound much more useful than json to me. Relatively few legacy command line programs handle JSON. For those that do, the terraform json encoding and decoding functions are present. For csv, yaml, base64, newline delimited text, and all of the other formats in common use, adding a json wrapper is unwelcome overhead.

[apparentlymart]

This provider was originally written before Terraform had a jsondecode function. I agree that with that function now present it would be better to have a new data source that can just run a program and capture its output as a string.

I would suggest doing that as a new data source in the local provider though, not as an extension of the external data source, because the external data source was designed for use with programs tailored to work with Terraform (as a middle-ground to allow writing small glue programs rather than entire new Terraform providers), not for executing arbitrary existing software with no modifications.

As an example, the new data source might look like this:

data "local_exec" "example" {
  program = ["example", "program", "generating", "csv"]
}

output "example" {
  value = csvdecode(data.local_exec.example.stdout)
}

As with external, it would be important to use this only for programs that don't have externally-visible side-effects, because the program would be run during the planning phase rather than the apply phase. But unlike external it would impose no constraints on the output except that it be UTF-8 encoded (because Terraform language strings are Unicode, not raw bytes) and leave the user to decide how and whether to parse it.

I don't work directly on either the external or the local providers, so I'm suggesting the above just as a design starting point, and I can't promise it would be accepted exactly like that. If you're interested in working on such a thing I'd suggest starting by opening an issue in the local provider repository to describe what you're intending to do and get feedback from the team that maintains that provider before doing work on it, in case there are design constraints for that provider that I'm not considering.

[AmudaPalani]

data "external" "subscription_quota_check" { depends_on = [null_resource.azlogin]

program = [ "az", "vm", "list-usage", "--location", local.cli_location, "--output", "json", "--query", "[?localName=='Total Regional vCPUs'].{Result:currentValue}" ] }

I am getting the same error:

Error: command "az" produced invalid JSON: json: cannot unmarshal array into Go value of type map[string]string

When I run the above command in CLI, i get the following output:

[ { "Result": "396" } ]

Is there any workaround to fix this in data "external"?

[rquadling]

@AmudaPalani Please take a look at https://github.com/digitickets/terraform-aws-cli as a way to handle this.

But .. possibly ... "[?localName=='Total Regional vCPUs'][0].{Result:currentValue}" may work.

[AmudaPalani]

I tried "[?localName=='Total Regional vCPUs'][0].{Result:currentValue}". Same error.

@rquadling Is your suggestion to write to output file and echo it? https://github.com/digitickets/terraform-aws-cli/blob/c3e4fa9d36da0a643b1350316e90969f511dcacc/scripts/awsWithAssumeRole.sh#L51

[rquadling]

I run run the AWS command with output as JSON to a file, if that fails, then I abort my script (which tells Terraform things didn't work out for us).

If you can get the exact value outputted via the command line (so a single value), then you can use the same query in terraform (either using the approach I built in my module or your own solution).

[rquadling]

As you are only interested in .currentValue, you don't need to wrap it in an object for it to be picked up! (I don't in my aws logic).

So, try "[?localName=='Total Regional vCPUs'][0].currentValue".

[AmudaPalani]

I tried

az vm list-usage --location "Central US" -o json --query "[?localName=='Total Regional vCPUs'].currentValue" [ "412" ]

Still same Error: command "az" produced invalid JSON: json: cannot unmarshal array into Go value of type map[string]string

[AmudaPalani]

Found it!

az vm list-usage --location "Central US" -o json --query "[?localName=='Total Regional vCPUs'].currentValue | [0]" Output:

"420"

[rquadling]

Ha. The JSON Querying language can be ... troublesome ... ! Well done though.

[AmudaPalani]

resource "null_resource" "azlogin" {

triggers = { always_run = timestamp() }

provisioner "local-exec" { command = "az login --service-principal -u ${var.client_id} -p ${var.client_secret} --tenant ${var.tenant_id}" }

}

data "external" "subscription_quota_check_cpu" {

depends_on = [null_resource.azlogin]

program = [ "az", "vm", "list-usage", "--location", local.cli_location, "--output", "json", "--query", "[?localName=='Total Regional vCPUs'].{Result:currentValue} | [0]" ]

}

When the above code is run first time, it works fine. When run second time, it throws this error:

Error: failed to execute az: ERROR: Please run 'az login' to setup account. on main.tf line 27, in data external subscription_quota_check_cpu: 27: data external subscription_quota_check_cpu {

I have azlogin login run every time. Any thoughts on how to fix this error?

[rquadling]

I've no idea on az (Azure I guess).

In AWS, one option is to set some environment variables and the AWS Terraform Provider and the AWS CLI can utilise them equally. There are several other approaches, but this particular pattern has worked well for us as we do not need to expose them in the repo. Our pipeline has secret variables (which only the senior dev team can configure).

If you have supplied credentials for the az Terraform Provider to use, how are they exposed such that az command has access to them?

[paololazzari]

For an example on how to return a JSON object with an external data source see this answer https://stackoverflow.com/questions/77139468/how-to-retrieve-s3-bucket-tags-using-data-source-in-terraform/77140460#77140460

[CarstenHS]

For the love of..

Please update the documentation with underlining the fact that the external data sourcing anno 2024 only supports the simplest json data in the shape of a simple map.

Furthermore please add an explicit and correctly shaped json example on what the resource needs.

Spent too much time figuring this out assuming we in 2024 and with data directly from az cli would be able to process this.