Feature Request - Allow list/array in `query` and `result`

[hashibot]

This issue was originally opened by @vikas027 as hashicorp/terraform#12249. It was migrated here as part of the provider split. The original body of the issue is below.

---

Terraform Version

0.8.7

Affected Resource(s)

external data source

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]

  query = {
    list_of_images = "${var.list_of_images}"
  }
}

Expected Behavior

List list_of_images should have been passed to the script.

Actual Behavior

Error refreshing state: 1 error(s) occurred:

* query: 1 error(s) decoding:

* '[list_of_images]' expected type 'string', got unconvertible type '[]interface {}'

---

Terraform Version

0.8.7

Affected Resource(s)

• external

Terraform Configuration Files

data "external" "create_policy" {
  program = ["bash", "${path.module}/create_policy.sh"]
}

create_policy.sh

#!/bin/bash
set -e
# This works
jq -n --arg foobaz "$FOOBAZ" '{"foobaz":$foobaz}'
# This throws an error
jq -n '{"Version":"2008-10-17","Statement":[{"Sid":"repo_policy","Effect":"Allow","Principal":{"AWS":["arn:aws:iam::${account_id}:root","arn:aws:iam::${account_id}:role/ecr_restricted_admin"]},"Action":"*"}]}'

Expected Behavior

Terraform shouldn't thrown an error when the the bash script is producing a valid json

$ bash modules/ecr/create_policy.sh  | jq .
{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "repo_policy",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::${account_id}:root",
          "arn:aws:iam::${account_id}:role/ecr_restricted_admin"
        ]
      },
      "Action": "*"
    }
  ]
}

Actual Behavior

* data.external.create_policy: command "bash" produced invalid JSON: json: cannot unmarshal array into Go value of type string

Steps to Reproduce

Create the resource and the script files and run terraform plan

[tomdavidson]

@apparentlymart Is this a viable universal work around?:

 query = {
    params = "base64endcode(jsonencode(var.list_of_images))"
  }

In the bash script use tf to decode:

Q=$(echo 'base64decode($PARAMS)' | terraform console) 

# do sripty stuff on Q

echo 'base64decode(jsonencode($OUTPUT))' | terraform console

[bgshacklett]

This workaround relies on bash being installed, meaning that it's not easily portable between macOS/Linux and Windows. In practice, I'm aware that Windows can run Bash, but depending on it for cross-platform compatibility is less than ideal.

In my personal use case, I've got a data structure which I'd like to pass directly to jq to keep the dependencies very lightweight. Most developers are likely to have jq on their systems so I'm much less bothered by requiring it as a dependency.

[casey-robertson]

Would really like this too. I want to build GCP custom roles by calling the gcloud CLI to grab permissions from built-in roles.... but for the life of me cannot get them back in a format TF is ok with. If it would just accept an array.

[karolinepauls]

Please rename the issue to include the result attribute. The issue request mentions both result and query in its contents. It would be a shame to get only one of the 2 cases mentioned fixed.

[bioshazard]

After being frustrated by the very limited JSON parsing capacity of the external provider, I found that jsondecode on a local file containing the same output works fine in a locals definition:

locals {
  # https://github.com/hashicorp/terraform/issues/12249
  # Ideally queried directly, but must be parsed locally.
  local_data = jsondecode(file("${path.module}/lib/test.json"))
  my_clusters = local.local_data.clusters

If the external provider simply parsed the stdout with the built-in jsondecode it would have the outcome I need.

[mattm-ecoatm]

Is this dead?

[apparentlymart]

At the time this was originally opened it was literally impossible for a provider to return values of a dynamic type selected based on the input, so I assume that's why this issue got stuck.

At this point it is technically possible (since Terraform v0.12) for a provider to announce that one of its result attributes has a dynamically-decided type, in which case the response from the provider is encoded to include in-band type information along with the value instead of just relying on the schema as would normally be the case.

However, I'm not sure if the plugin framework has full support for that capability at this time. The Dynamic Type documentation looks promising, but I've never tried to use those features so I don't know if it's sufficient for implementing what was requested here.