Add examples of custom variable and function lookups

[paulspringett]

From the conversation in https://github.com/hashicorp/hil/pull/8

It would be very useful to implement lookup functions for variables and functions without modifying HIL package code.

One example use-case would be reading arbitrary environment variables, eg.

greeting = "Hi ${env.NAME}"

This initially appeared possible by passing something that implements the Scope interface to the EvalConfig function, however other parts of the code explicitly reference the BasicScope implementation making this not possible.

If this is already possible it would be great to document some examples in the README. If not would it be possible to allow other implementations of the Scope interface?

[mitchellh]

Thanks. We do this sort of thing (not env vars but similar) in other projects. It isn't obvious how to do it and even within HashiCorp I show some people and there is surprise that its possible. I'll try to doc this!

[paulspringett]

Hey @mitchellh any chance you're able to document this? If not a pointer in the right direction would be awesome (eg. links to an example or source code). Thanks!

[paulspringett]

@mitchellh any chance of an update on this? If it's not possible feel free to close the issue.

[pburkholder]

@mitchellh +1 on documentation here.

[apparentlymart]

Hi @paulspringett,

I'm not sure what exactly @mitchellh was going to document here (he certainly knows a lot more about this subject than I do!) but my understanding of how Terraform deals with this is as follows:

• Pass the input interpolation string into hil.Parse to get back a HIL AST root node.
• Use the ast "Visitor" interface to walk the tree looking for *ast.VariableAccess nodes, and extract their names.
• Once you've got the set of names, do whatever you need to do in order to parse the names and compute values for them. In your case, you'd presumably be looking for names that start with env..
• Put the values of these names into the BasicScope variable map before you call Eval, so that they will be there when needed.

This is a few more steps than I imagine you were expecting. Consider that HIL was originally written for Terraform and Terraform wants to be able to do as much validation as possible before taking any action. Thus this separation of "discover variables" and "evaluate" allows Terraform to ensure that all of the variables are valid before any Interpolation is done, which is sometimes before the values of the variables are even known.

I honestly had the same question as you r.e. why there's a Scope interface but yet some HIL functions take *ast.BasicScope directly as arguments. I don't know the answer to that, but the above pattern allows you to work around it.

[purpleidea]

@apparentlymart There isn't an obvious example of how to Walk the tree using Visitor in the docs, but if you've got one, please let us know :) I'm fooling around with this now.

[apparentlymart]

Hi @purpleidea,

Terraform's implementation of this is DetectVariables, which I think is about as simple as this can be. Terraform uses a type called InterpolatedVariable to represent the different variable prefixes it deals with so that the actual lookup can be deferred until the graph walk, so if your data is already available then you could potentially simplify this by directly populating the variable table as part of the walk.

---

Depending on the nature of your current work, you may find it interesting to look at HCL2, which is a currently-experimental new version of HCL that incorporates HIL functionality to form a single language. It's not yet final and so it's subject to change as we continue to develop it, but if you're early in your app development and/or tolerant of a little instability, you may find the API of this new version more convenient: after parsing an expression, the Expression object has a Variables method that does the necessary tree walking internally, to return a list of what HCL2 calls "traversals". HIL's implementation is definitely more battle-tested though, despite the less-convenient API.

[purpleidea]

@apparentlymart This was very helpful information, thank you.

I'm actually currently only interested in the hil part, since I wanted to avoid implementing a string interpolation library for the https://github.com/purpleidea/mgmt/ language (WIP), and it seems to be a great fit so far.

Do you think the string interpolation part of HCL2 (without the rest of the language) is easy to call out of that library instead? a grep -i finds no mention of interpolation. I don't care about API breaks, as long as they're not gratuitous and that git master builds, so happy to help bang on that if it's preferable to you folks.

I'm also especially curious to have the discussion about why you all switched from go-yacc to pure golang, but perhaps that's better had out of this issue. In case you'd like to chat more, I'm in #mgmtconfig on Freenode IRC.

Thanks again!

[apparentlymart]

Hi @purpleidea!

At the moment HIL is still in heavy use in HashiCorp products such as Terraform, so we'll be continuing to maintain it until HCL2 reaches the point where it's stable and being used "for real" in products. If you're happy with its current functionality then it's fine to keep using HIL, especially if you plan to vendor it.

In HCL2 the idea of "interpolation" has generalized to instead be called "templates", so that's why the word "interpolation" doesn't turn up anything interesting there. The HIL-equivalent functionality in HCL is the ParseTemplate function. Terraform is currently being used as the test bed for building out HCL2, so we may change things based on implementation needs and user feedback; syntax details -- particularly things that are new relative to HIL, like the template directives -- are subject to change if we find shortcomings during this experimental phase.

You can find some details on the HIL scanner/parser rewrite in #32. HCL2 is actually currently using a generated scanner with a hand-written parser, since the merged grammar got complex enough that the hand-written scanner grew unwieldy. If you do decide to work with HCL2 templates and have feedback on it, issues in its own repository are very welcome; the design is intentionally constrained to be broadly compatible with existing HCL and HIL usage, but there are several new features.

[purpleidea]

@apparentlymart I got all my HIL stuff working so I'm going to stick with that for now, but happy to try and port things over in the future. Thanks again for your comments! Feel free to ping if you'd ever like to see the mgmt stuff.