TFJSON incompatibilities with HCL

[eropple]

So I've been bouncing around the mailing list and the IRC channel for a couple days, trying to figure out the JSON format for Terraform. This isn't intended as a support request, because I legitimately don't know if things are working as intended; I'm concerned that there are actual bugs here and I'd like some help sussing that out.

I've written a tool to put a DSL on top of Terraform for some logic that I personally need in my provisioning layer. Which is fine, Terraform ostensibly reads JSON, everything's cool. The problem is that Terraform does not parse the output of hcl2json and there appears to be no documentation on proper TFJSON structure. Maybe this should be two separate bugs, but they feel so interlinked as to have the latter be fallout from the former.

Terraform does not parse the output of hcl2json

I should qualify this by saying that I've investigated this to the best of my abilities right now and avoided filing a bug against Terraform until I'd checked out other alternatives. The developer of hcl2json seems quite sure that the application in the hcl2json pull request correctly emits JSON as per HCL's correct behaviors, so I'm filing it here.

Example follows, using Terraform 0.3.6 and the examples from the master branch:

[ed ~/Development//terraform/examples/aws-count (master)]$ ls
README.md  main.tf  outputs.tf  variables.tf
[ed ~/Development/terraform/examples/aws-count (master)]$ mkdir test
[ed ~/Development/terraform/examples/aws-count (master)]$ ~/Development//hcl/cmd/hcl2json/hcl2json main.tf > test/main.tf
[ed ~/Development/terraform/examples/aws-count (master ✗)]$ ~/Development/hcl/cmd/hcl2json/hcl2json variables.tf > test/variables.tf
[ed ~/Development/terraform/examples/aws-count (master ✗)]$ cd test
[ed ~/Development/terraform/examples/aws-count/test (master ✗)]$ ~/Downloads/terraform_0.3.6_darwin_amd64/terraform plan
Error loading config: Error loading /Users/Ed/Development/terraform/examples/aws-count/test/main.tf: Error reading config for [aws_elb]: root: not an object type for map (6)
[1]    14833 exit 1     ~/Downloads/terraform_0.3.6_darwin_amd64/terraform plan

aws_elb is the first item under resource in the first file; something looks tremendously un-copacetic here. @josephholsten might be mistaken in his estimation about TF, but I'd appreciate some clarity and some guidance if so.

there appears to be no documentation on proper TFJSON structure

My issues with hcl2json that led to part 1 arose after @mitchellh suggested its use to decompose an HCL document to figure out the correct JSON to generate and emit for provisioner in TFJSON. As near as I can tell, there's no documentation for TFJSON, and my understanding from the above is that HCL itself is not completely canonical in all respects? It would be really helpful to see an actually working, nontrivial example of TFJSON somewhere, using the features of Terraform, to pattern after.

.

I really really really dig Terraform and I think there's a lot of potential here, but I'm not going to be able to sell my employer on Terraform (and by extension Atlas, which I'd like to use and have us give you money for, because it will save us tons of time!) if I spend much more time spinning my wheels--I wrote Terraframe because I'm 100% comfortable doing the work to make our edge-ish cases work, but I've been swimming upstream to use Terraform's advertised features and there's only so much leash I have to explore this option. I'm getting close to the "this needs to work or I need to bail on Terraform" choice, and I'm most grateful for any help y'all can provide to help me stump for the HashiCorp ecosystem. =)

[mitchellh]

I've reproduced this. One provisioner works but getting multiple seems to not. I'll take a look at HCL since I think the bug is in there.

[josephholsten]

BTW, I'm not 100% sure hcl2json is doing the right thing. It's just the simplest impl I could think of: hcl:cmd/hcl2json/main.go:53-63

Without the error handing and I/O, it's doing this:

d, _ := ioutil.ReadAll(c.Stdin)
var obj interface{}
_ = hcl.Decode(&obj, string(d))
res, _ := json.MarshalIndent(obj, "", " ")
fmt.Println(string(res))

[mitchellh]

This isn't an issue with hcl2json. I think the issue if anything is core to HCL but might be easier in the short term to patch in Terraform.

[mitchellh]

Fixed, I'm going to work on better documenting but for now view that test case.

[renier]

@mitchellh I'm not sure that the problem is fixed. I don't get an error from plan when loading a JSON .tf file, but it doesn't find anything to do vs. loading the HCL .tf version:

renier@taurus test $ cat test.hcl
provider "softlayer" {}

resource "softlayer_ssh_key" "my_key" {
  label = "Renier Key Add Test"
  public_key = "${file("~/.ssh/id2_rsa.pub")}"
}

renier@taurus test $ hcl2json test.hcl test.tf

renier@taurus test $ cat test.tf
{
    "provider": [
        {
            "softlayer": [
                {}
            ]
        }
    ],
    "resource": [
        {
            "softlayer_ssh_key": [
                {
                    "my_key": [
                        {
                            "label": "Renier Key Add Test",
                            "public_key": "${file(\"~/.ssh/id2_rsa.pub\")}"
                        }
                    ]
                }
            ]
        }
    ]
}

renier@taurus test $ ls *.tf
-rw-r--r--  1 renier  staff   480B Oct 11 16:49 test.tf

renier@taurus test $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

No changes. Infrastructure is up-to-date. This means that Terraform
could not detect any differences between your configuration and
the real physical resources that exist. As a result, Terraform
doesn't need to do anything.

renier@taurus test $ cp test.hcl test.tf

renier@taurus test $ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

+ softlayer_ssh_key.my_key
    fingerprint: "<computed>"
    label:       "Renier Key Add Test"
    public_key:  "ssh-rsa ********\n"

Plan: 1 to add, 0 to change, 0 to destroy.

[oogali]

@renier Can you open a new issue if you're still having this problem?

[renier]

@oogali I did #9354

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.