Article example doesn't work

[scottf]

Author wrote an article some time ago: http://jen20.com/2015/09/07/using-hcl-part-1.html

Granted maybe there is stuff missing from the article like configuration. Are there ANY docs for this library?

Expected behavior

It works like it says in the article

Actual behavior

The data is only populated when words are not camel cased. `&{us-west-2 backups []}'

Steps to reproduce

Here is my full code:

package main

import (
    "github.com/hashicorp/hcl"
    "fmt"
)

type Config struct {
    Region      string
    AccessKey   string
    SecretKey   string
    Bucket      string
    Directories []DirectoryConfig
}

type DirectoryConfig struct {
    Name                  string
    SourceDirectory       string
    DestinationPrefix     string
    ExcludePatterns       []string
    PreBackupScriptPath   string
    PostBackupScriptPath  string
    PreRestoreScriptPath  string
    PostRestoreScriptPath string
}

func main() {
    hclParseTree, err := hcl.Parse(data)
    if err != nil {
        fmt.Println("2", err)
    } else {
        cfg := &Config{}
        err := hcl.DecodeObject(cfg, hclParseTree)
        if err != nil {
            fmt.Println("3", err)
        } else {
            fmt.Println("4", cfg)
        }
    }
}

var data = "region = \"us-west-2\"\n" +
    "access_key = \"something\"\n" +
    "secret_key = \"something_else\"\n" +
    "bucket = \"backups\"\n" +
    "\n" +
    "directory \"config\" {\n" +
    "    source_dir = \"/etc/eventstore\"\n" +
    "    dest_prefix = \"escluster/config\"\n" +
    "    exclude = []\n" +
    "    pre_backup_script = \"before_backup.sh\"\n" +
    "    post_backup_script = \"after_backup.sh\"\n" +
    "    pre_restore_script = \"before_restore.sh\"\n" +
    "    post_restore_script = \"after_restore.sh\"\n" +
    "}\n" +
    "\n" +
    "directory \"data\" {\n" +
    "    source_dir = \"/var/lib/eventstore\"\n" +
    "    dest_prefix = \"escluster/a/data\"\n" +
    "    exclude = [\n" +
    "        \"*.merging\"\n" +
    "    ]\n" +
    "    pre_restore_script = \"before_restore.sh\"\n" +
    "    post_restore_script = \"after_restore.sh\"\n" +
    "}"

[sethvargo]

Hi @scottf

This is the way most parsing libraries work in go - they assume the field is named the same. If you are using underscore names, you'll need to label the struct fields as so:

type Config struct {
    Region      string
    AccessKey   string `hcl:"access_key"`
    SecretKey   string `hcl:"secret_key"`
    Bucket      string
    Directories []DirectoryConfig
}

[scottf]

Thanks for the response, that works, sorta. So I continued to try to understand and coded something up. The parse flat out does not work for sub structures. Here is the new code for scrutiny, look at the very end results. The object ends up with 4 Bar objects (there are only 3 in the data), and the only one that is correct is the one where I put name both inside and outside bar "nameB" {name = "nameBprime"

package main

import (
    "github.com/hashicorp/hcl"
    "fmt"
)

type Foo struct {
    Region      string
    CamelCase   string `hcl:"camel_case"`
    Bars        []Bar  `hcl:"bar"`
}

type Bar struct {
    Name      string
    CamelBaz  string   `hcl:"camel_baz"`
}

func main() {
    fmt.Println(data, "\n")
    cfg := &Foo{}
    err := hcl.Decode(cfg, data)
    if err != nil {
        fmt.Println("Err", err)
    } else {
        fmt.Println(fmt.Sprintf("Foo: Region='%s' CamelCase='%s' Bars? %d", cfg.Region, cfg.CamelCase, len(cfg.Bars)))
        for _,bar := range cfg.Bars {
            fmt.Println(fmt.Sprintf("    Bar: Name='%s' Baz='%s'", bar.Name, bar.CamelBaz))
        }
    }
}

var data = "region = \"us-west-2\"\n" +
    "camel_case = \"blah\"\n" +
    "\n" +
    "bar \"nameA\" {\n" +
    "    camel_baz = \"bazA\"\n" +
    "}\n" +
    "bar \"nameB\" {\n" +
    "    name = \"nameBprime\"\n" +
    "    camel_baz = \"bazB\"\n" +
    "}\n" +
    "bar {\n" +
    "    name = \"nameCprime\"\n" +
    "    camel_baz = \"bazC\"\n" +
    "}"

The output is this:

region = "us-west-2"
camel_case = "blah"

bar "nameA" {
    camel_baz = "bazA"
}
bar "nameB" {
    name = "nameBprime"
    camel_baz = "bazB"
}
bar {
    name = "nameCprime"
    camel_baz = "bazC"
} 

Foo: Region='us-west-2' CamelCase='blah' Bars? 4
    Bar: Name='' Baz='bazA'
    Bar: Name='nameBprime' Baz='bazB'
    Bar: Name='nameCprime' Baz=''
    Bar: Name='' Baz='bazC'

[sethvargo]

Hey @scottf

Thank you for your reply. It's probably best if you print out the JSON instead of using a custom dump function, as it'll be clearer what the library is doing:

package main

import (
    "encoding/json"
    "fmt"

    "github.com/hashicorp/hcl"
)

type Foo struct {
    Region    string
    CamelCase string `hcl:"camel_case"`
    Bars      []Bar  `hcl:"bar"`
}

type Bar struct {
    Name     string
    CamelBaz string `hcl:"camel_baz"`
}

func main() {
    var cfg Foo
    if err := hcl.Decode(&cfg, data); err != nil {
        panic(err)
    }

    b, err := json.MarshalIndent(cfg, "", "  ")
    if err != nil {
        panic(err)
    }
    fmt.Printf("%s\n", b)
}

var data = `
region     = "us-west-2"
camel_case = "blah"

bar "nameA" {
  camel_baz = "bazA"
}

bar "nameB" {
  name      = "nameBprime"
  camel_baz = "bazB"
}

bar {
  name      = "nameCprime"
  camel_baz = "bazC"
}
`

In short, because you did not provide a string to the third bar function, there's no way to safely merge that data, so it's split into two objects. If you supply a name to the third bar, you get three bars as expected.

Since you're "naming" the bars, you probably want to decode into a map[string]*Bar, not []Bar, which will give you this:

{
  "Region": "us-west-2",
  "CamelCase": "blah",
  "Bars": {
    "nameA": {
      "Name": "",
      "CamelBaz": "bazA"
    },
    "nameB": {
      "Name": "nameBprime",
      "CamelBaz": "bazB"
    },
    "nameC": {
      "Name": "nameCprime",
      "CamelBaz": "bazC"
    }
  }
}

[scottf]

The json helps clear it up and that's a much easier way to put the data in the code. It still doesn't work as I expected for that data, there are still 4 bars. As for the []Bar, that was how the example in the original article did it, and that's what I tested with.

It seems that you must name (tag?) the object, but the name can be empty.

bar "" {
    name = "nameAprime"
    camel_baz = "bazA"
}
bar "" {
    name = "nameBprime"
    camel_baz = "bazB"
}

The other two formats don't work at all, I tested them individually. Thanks for your help.

[sethvargo]

Hi @scottf

You're reading an article dated 2015, so I'm not surprised there are differences now. HCL was just a baby in 2015 and has since matured greatly. Can you tell me what your expected output is (in JSON), and I can help you write the equivalent HCL?

[client9]

update with massive edit. update 3: more corrections

Hi @scottf - I found the terraform HCL doco to be useful. https://www.terraform.io/docs/configuration/syntax.html

Also this function: just dump in your HCL file and it pops out the equiv JSON

func MustDecode(config string) {
        var obj interface{}
        if err := hcl.Decode(&obj, config); err != nil {
                log.Printf("Unable to decode: %s", err)
        }
        out, err := json.MarshalIndent(obj, "", " ")
        if err != nil {
                log.Fatalf("Marshal fail: %s", err)
        }
        fmt.Println(string(out))
}

So this works "as expected"

filter {
  include = "*"
}
filter {
  include = "*.png"
}

Here's the JSON

{
 "filter": [
  {
   "include": "*"
  },
  {
   "include": "*.png"
  }
 ]
}

great!

Now let's nest it

root filter {
  include = "*"
}
root filter {
  include = "*.png"
}

and the json is...

{
 "root": [
  {
   "filter": [
    {
     "include": "*"
    }
   ]
  },
  {
   "filter": [
    {
     "include": "*.png"
    }
   ]
  }
 ]
}

Since the root object automatically makes lists (even with same name), these two stanza do not get merged, and are completely separate objects.

Hope this helps (round 3)

n