[Question] How do I handle the following fields in the golang code？

[guodongq]

I would like to generate the following file.

locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  region      = local.region_vars.locals.region

  project        = "abc-dev"
  account_number = "443813933915"
  services = [
    "apiserver",
  ]
  additional_statements = [
    {
      "Sid" : "",
      "Effect" : "Allow",
      "Action" : [
        "secretsmanager:GetSecretValue",
        "kms:Decrypt"
      ],
      "Resource" : [
        "arn:aws:secretsmanager:${local.region}:${local.account_number}:secret:*",
        "arn:aws:kms:${local.region}:${local.account_number}:key/*"
      ]
    }
  ]
}

But in my golang code, I can define a structure as following

type ProjectHcl struct {
    Locals Locals `hcl:"locals,block"`
}

type Locals struct {
    Project       string   `hcl:"project"`
    AccountNumber string   `hcl:"account_number"`
    Services      []string `hcl:"services"`
}

How can I deal with other fields region_vars, region, additional_statements?

[apparentlymart]

Hi @guodongq,

By default, gohcl will produce an error if there are any arguments in the block that don't correspond to fields you've declared in the structure.

If you want to ignore the extra arguments or to process them separately later, you can add an extra field to Locals to capture the "remaining" content from that block:

type Locals struct {
    Project       string   `hcl:"project"`
    AccountNumber string   `hcl:"account_number"`
    Services      []string `hcl:"services"`
    Remain        hcl.Body `hcl:",remain"`
}

gohcl treats that ",remain" tag in a special way, gathering anything that wasn't already decoded into one of the other fields into another hcl.Body object which you can either ignore completely or use for more decoding later.

If your goal here is to mimic Terraform's handling of a locals block, where the content of locals is just an arbitrary collection of attributes, you can alternatively set the type of field Remain to be hcl.Attributes instead of hcl.Body. hcl.Attributes is a map from argument name to hcl.Expression, and so it allows you to dynamically react to whatever attributes are written, rather than having to specify specific attribute names in the schema.

[guodongq]

@apparentlymart Thank you for your help

I'm having problems with the following scenario， I write a unit test as following:

import (
    "fmt"
    "github.com/hashicorp/hcl/v2"
    "github.com/hashicorp/hcl/v2/gohcl"
    "github.com/hashicorp/hcl/v2/hclsimple"
    "github.com/hashicorp/hcl/v2/hclwrite"
    "testing"
)

func TestHCL(t *testing.T) {
    const hclStr = `
locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  region      = local.region_vars.locals.region

  project        = "abc-dev"
  account_number = "443813933915"
  services = [
    "apiserver",
  ]
  additional_statements = [
    {
      "Sid" : "",
      "Effect" : "Allow",
      "Action" : [
        "secretsmanager:GetSecretValue",
        "kms:Decrypt"
      ],
      "Resource" : [
        "arn:aws:secretsmanager:${local.region}:${local.account_number}:secret:*",
        "arn:aws:kms:${local.region}:${local.account_number}:key/*"
      ]
    }
  ]
}
`

    type Locals struct {
        Project       string         `hcl:"project"`
        AccountNumber string         `hcl:"account_number"`
        Services      []string       `hcl:"services"`
        Remain        hcl.Attributes `hcl:",remain"`
    }

    type ProjectHcl struct {
        Locals Locals `hcl:"locals,block"`
    }

    var projectHCL ProjectHcl
    err := hclsimple.Decode("project.hcl", []byte(hclStr), nil, &projectHCL)
    if err != nil {
        t.Fatal(err)
    }

    // add new project's service
    projectHCL.Locals.Services = append(projectHCL.Locals.Services, "testabc")

    hclFile := hclwrite.NewEmptyFile()
    gohcl.EncodeIntoBody(&projectHCL, hclFile.Body())

    fileBytes := hclFile.Bytes()
    file := string(fileBytes)
    fmt.Println(file)
}

You can see that a structure projectHCL decoded from a HCL string, and I add new values testabc into projectHCL.Locals.Services.

Finally I'm going to regenerate the HCL string, but I find that fields region_vars, region, additional_statements are all missing.

Is there something wrong with my HCL encode code? Could you please give me a little idea, thanks

[apparentlymart]

Hi @guodongq,

gohcl is not the appropriate level of abstraction for making this sort of partial modification to an existing file, because the decoding and re-encoding process will both lose some information that isn't normally needed for typical application use of the configuration.

Instead, you should implement your editing code entirety using the hclwrite package which can work directly with the physical syntax tree of the source file. If you parse using hclwrite's own parser then you will get an hclwrite.File that already contains the original file content, and then you can use the hclwrite API to find the locals block you want to modify and then call SetAttributeValue on its body to change that file directly.

If you then write the modified file back to bytes again you will see that it preserved all of the other content in the file, including any comments, and should have changed only the attribute you specified and possibly the indentation and spacing between tokens.

[guodongq]

Hi @apparentlymart

I change the code as you explained, the code as following:

func TestHCL(t *testing.T) {
    const hclStr = `
locals {
  region_vars = read_terragrunt_config(find_in_parent_folders("region.hcl"))
  region      = local.region_vars.locals.region

  project        = "abc-dev"
  account_number = "443813933915"
  services = [
    "apiserver",
  ]
  additional_statements = [
    {
      "Sid" : "",
      "Effect" : "Allow",
      "Action" : [
        "secretsmanager:GetSecretValue",
        "kms:Decrypt"
      ],
      "Resource" : [
        "arn:aws:secretsmanager:${local.region}:${local.account_number}:secret:*",
        "arn:aws:kms:${local.region}:${local.account_number}:key/*"
      ]
    }
  ]
}
`

    f, diags := hclwrite.ParseConfig([]byte(hclStr), "project.hcl", hcl.Pos{Line: 1, Column: 1})
    if diags.HasErrors() {
        fmt.Printf("errors: %s", diags)
        return
    }

    body := f.Body()

    for _, v := range body.Blocks() {
        blockBody := v.Body()

        // services
        attr := blockBody.GetAttribute("services")
        if attr != nil {
            fmt.Printf("got it: %#v\n", attr)

                        // can not get the old services's values from this attr
                var oldServices []cty.Value 
                blockBody.SetAttributeValue("services", cty.ListVal(append(oldServices, cty.StringVal("abc"))))

        }

    }

    bytes := f.Bytes()
    fmt.Println(bytes)
    fmt.Println(string(bytes))
}

But I can't get the oldServices from attr, and new values about the services should contain the old value.