Unnamed blocks?

[andrenth]

I have the following type definitions:

type Config struct {
    Init string
    Host []HostConfig `hcl:"host"`
}

type HostConfig struct {
    Services []Service `hcl:"service"`
}

type Service struct {
    Name      string `hcl:"name,key"`
    PidFile   string `hcl:"pid_file"`
}

This allows me to parse the following hcl:

init = "systemd"

host "foo" {
  service "apache2" {
    pid_file  = "/var/run/apache2.pid"
  }
  service "ssh" {
    pid_file  = "/var/run/sshd.pid"
  }
}

However the string that follows host is useless for me. Is it possible to parse a config file without it, i.e.

host {
  ...
}

Thanks.

[apparentlymart]

Hi @andrenth,

With the types you showed there, I would in fact expect the HCL parser to allow the unlabeled block form you show in your final example here.

Can you share a little more detail on what happened when you attempted this? For example, any error messages you saw, or how the actual result differed from what you expected.

[andrenth]

Hello Martin

Here's a minimal program:

package main

import (
    "fmt"
    "io/ioutil"
    "log"
    "os"

    hcl "github.com/hashicorp/hcl"
)

type Config struct {
    Host []HostConfig `hcl:"host"`
}

type HostConfig struct {
    Name     string    `hcl:"name,key"`
    Services []Service `hcl:"service"`
}

type Service struct {
    Name    string `hcl:"name,key"`
    PIDFile string `hcl:"pid_file"`
}

func main() {
    var config Config

    contents, err := ioutil.ReadFile(os.Args[1])
    if err != nil {
        log.Fatal(err)
    }
    err = hcl.Decode(&config, string(contents))
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(config)
}

With the config file below, the output is {[{myhost [{apache2 /var/run/apache2/apache2.pid} {ssh /var/run/sshd.pid}]}]}, as expected. If I remove "myhost" from it, i.e. just use host { ... }, then it becomes {[{service []} {service []}]}.

host "myhost" {
  service "apache2" {
    pid_file  = "/var/run/apache2/apache2.pid"
  }
  service "ssh" {
    pid_file  = "/var/run/sshd.pid"
  }
}

[apparentlymart]

Hi @andrenth,

Your struct HostConfig has a field with a struct tag containing key, which causes the decoder to expect a label there.

The decoder is actually operating at a lower level of abstraction where the input was already converted into a tree of key/value "objects", so it uses "key" to mean that label, and so if you specify key but then don't provide one it will treat the nested blocks as the keys, as if you'd written the following:

host "service" {
}
host "service" {
}

Because these blocks don't then contain another key service, the list of services appears as empty in the output, as you saw.

A key to understanding this behavior is that the decoder treats the HCL syntax as just syntactic sugar for JSON, and so your example configuration file is internally interpreted as if you wrote the following JSON

{
  "host": {
    "myhost": {
      "service": {
        "apache2": {
          "pid_file": "/var/run/apache2/apache2.pid"
        },
        "ssh": {
          "pid_file": "/var/run/sshd.pid"
        }
      }
    }
  }
}

By removing the "myhost" label, you've effectively removed one level of hierarchy from this structure as far as the decoder is concerned, and so everything beneath gets misinterpreted.

With all of that background information in mind, you have a few options here:

• If you remove key from that struct tag then the decoder should interpret your version without "myhost" as you wanted. However, if the user does include the "myhost" label then the decoder will probably interpret it in some other strange way.

• You can skip using the decoder and work directly with HCL's AST, which will then allow you to recognize the distinctions between block types, block labels, and attribute names, at the expense of dealing with a lower-level API. This approach usually means that the JSON serialization of HCL won't work properly, however.

• You could instead use the next-generation version of HCL, which is still under development but I think should have enough functionality for what you are trying to do here.

---

Here's an equivalent of your simple program here using the HCL2 APIs instead:

package main

import (
    "log"
    "os"

    "github.com/davecgh/go-spew/spew"
    "github.com/hashicorp/hcl2/gohcl"
    "github.com/hashicorp/hcl2/hcl"
    "github.com/hashicorp/hcl2/hclparse"
)

type Config struct {
    Host []HostConfig `hcl:"host,block"`
}

type HostConfig struct {
    Name     string    `hcl:"name,label"`
    Services []Service `hcl:"service,block"`
}

type Service struct {
    Name    string `hcl:"name,label"`
    PIDFile string `hcl:"pid_file,attr"`
}

func main() {
    var config Config
    var diags hcl.Diagnostics

    parser := hclparse.NewParser()
    file, parseDiags := parser.ParseHCLFile(os.Args[1])
    diags = append(diags, parseDiags...)
    if diags.HasErrors() {
        log.Fatal(diags)
    }

    decodeDiags := gohcl.DecodeBody(file.Body, nil, &config)
    diags = append(diags, decodeDiags...)
    if diags.HasErrors() {
        log.Fatal(diags)
    }

    spew.Dump(config)
}

If I pass it your example with the "myhost" label:

(main.Config) {
 Host: ([]main.HostConfig) (len=1 cap=1) {
  (main.HostConfig) {
   Name: (string) (len=6) "myhost",
   Services: ([]main.Service) (len=2 cap=2) {
    (main.Service) {
     Name: (string) (len=7) "apache2",
     PIDFile: (string) (len=28) "/var/run/apache2/apache2.pid"
    },
    (main.Service) {
     Name: (string) (len=3) "ssh",
     PIDFile: (string) (len=17) "/var/run/sshd.pid"
    }
   }
  }
 }
}

Since the HCL2 decoder (gohcl) operates at the AST level rather than lowering to JSON first, it will detect and report situations where the input is improperly structured, such as if I remove the "myhost" label from the host block:

2018/06/18 17:41:54 simple-hcl2-api-example/try.hcl:1,6-7: Missing name for host; All host blocks must have 1 labels (name).

After removing the Name field from the HostConfig struct altogether:

type HostConfig struct {
    Services []Service `hcl:"service,block"`
}

...I get the result I think you wanted here:

(main.Config) {
 Host: ([]main.HostConfig) (len=1 cap=1) {
  (main.HostConfig) {
   Services: ([]main.Service) (len=2 cap=2) {
    (main.Service) {
     Name: (string) (len=7) "apache2",
     PIDFile: (string) (len=28) "/var/run/apache2/apache2.pid"
    },
    (main.Service) {
     Name: (string) (len=3) "ssh",
     PIDFile: (string) (len=17) "/var/run/sshd.pid"
    }
   }
  }
 }
}

...and if I put back the "myhost" label in configuration, it reports the opposite error:

2018/06/18 17:44:39 simple-hcl2-api-example/try.hcl:1,6-14: Extraneous label for host; No labels are expected for host blocks.

We're currently in the process of moving Terraform over to this new implementation, and so some details might change along the way if we find some issues, but in practice we haven't needed to make any changes for a while now so if you're willing to deal with the potential for some small API changes moving forward then you could make use of the HCL2 packages via vendoring for now.

Along with the improved robustness of the decoder, those hcl.Diagnostics values contain more information than you'd get from an error in the current HCL, so you can produce nicer error messages if you use NewDiagnosticTextWriter: