Generating multiple similar files from the same template

[FlorinAsavoaie]

I happened to need this several times until now and I'm pretty sure I am not the only one. A few ways I see it to be implemented:

1. Add a writeToFile function in the template. This could be used in conjunction with nested templates and get the thing going really fast, probably also very fast to implement (would be more difficult to write tests I guess than actually implement it). Not sure what would be the biggest downsides of this.

2. Add a template_set functionality in the configuration file. The biggest problem I see is that usually you want to generate this kind of templates by looping through a list of keys. This would introduce a "first" in the consul-template code, as the engine would have to implement logic outside the template to do this kind of looping according to some pattern presented by the user.

If we agree on a design together, I can work on it and submit a PR myself.

Thanks,

[FlorinAsavoaie]

@slackpad @sethvargo @bensojona ... Sorry for pinging you guys but I'd appreciate some feedback on this. I'm really interested to contribute with such a feature but I wouldn't want to work in vain.

[sethvargo]

Hi @FlorinAsavoaie,

I am no longer employed at HashiCorp, so I cannot help you out. I think @slackpad may have also left the company. Sorry!

[bensojona]

I’ll ask around internally and get back to you.

On Sun, Feb 25, 2018 at 19:13 Seth Vargo notifications@github.com wrote:

Hi @FlorinAsavoaie https://github.com/florinasavoaie,

I am no longer employed at HashiCorp, so I cannot help you out. I think @slackpad https://github.com/slackpad may have also left the company. Sorry!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/hashicorp/consul-template/issues/1077#issuecomment-368378309, or mute the thread https://github.com/notifications/unsubscribe-auth/ACUWauJgQyQCPpswflAIIPC9m2ofjBL2ks5tYiE9gaJpZM4SJqcg .

-- Jon

[FlorinAsavoaie]

Hey @bensojona ,

Any news on this, please?

Thanks, Florin.

[FlorinAsavoaie]

I thought a bit some more on this and there are multiple problems that I see in implementing a template_set directive:

1. Generating the list of destinations. Maybe using another template for this that renders a filename per line, which we consume at the other end of the buffer into a slice? We'd also need to keep the old version until the new one is up so we can delete the files no longer present.

2. Creating a 'template' for each destination. Might not be that complicated once we have the list of destinations.

3. Passing information about the destination to the template itself (so it knows what to render). For this one, I think we can already make a first step in adding a function like "templateDestination" that would return the filename to be written to the disk. I already looked into doing this (it would have more use-cases) but that information is not available anywhere around the template execution context, it's not a trivial change, I think.

Other issues:

• How does this interact with -dry?

[bensojona]

@FlorinAsavoaie I've pinged the Consul team again to reach out, apologies!

[banks]

Hi @FlorinAsavoaie

I'm some way behind you in familiarity with consul-template at this point - some of the main contributors deeply familiar with the code here have recently moved on to new jobs as you've discovered in this thread.

Could you provide a little more colour for what you are trying to achieve with this design? A concrete example or two of how you might want to use this would help build some context.

Thanks

[FlorinAsavoaie]

Hi @banks,

The most easy example is getting a list of certificates from Vault. A potential example configuration could be something like this:

consul-template configuration:

template_set {
  source = "certificate.pem.ctmpl"
  destination_prefix = "/etc/nginx/certificates/"
  destination_suffix = ".pem"
  destinations = "{{ range secrets "/my-certificates/" }}.{{ end }}"
}

certificate.pem.ctmpl:

{{ with secret (printf "/my-certificates/%s" destination) }}
{{ .private_key }}
{{ .certificate }}
{{ end }}

Where destination is a function that returns the "current" value of the destinations range, for the current template.

This would get a list of the certificates that we have stored in Vault, as secrets, and write them on the disk. There's currently no easy way to do this (right now I have a "hack" where I send them all to the same file using a separator and split them with a script in "command". However, we're also looking into using this together with nomad, and that makes it even more complicated.

Does this make sense?

Again, if we agree on the design, I can come up with an implementation myself, I hope.

[FlorinAsavoaie]

3. Passing information about the destination to the template itself (so it knows what to render). For this one, I think we can already make a first step in adding a function like "templateDestination" that would return the filename to be written to the disk. I already looked into doing this (it would have more use-cases) but that information is not available anywhere around the template execution context, it's not a trivial change, I think. - @FlorinAsavoaie

I think the easiest and most non-invasive way would be to implement a fake environment variable available in the template, like CONSUL_TEMPLATE_DESTINATION. This can be easily added in the Runner, Line 664. Would this be acceptable?

[FlorinAsavoaie]

@banks, any thoughts?

Thanks,

[FlorinAsavoaie]

Yeah, well, not so easy after all. In order to make things more efficient (although I totally don't understand why this is needed - why would you generate the same file over and over again, on the same system), consul-template has a very interesting approach of not rendering a template more than once if the template source code has the same MD5 as another template source code.

I wonder what would be an official solution from Hashicorp to get a list of certificates (without knowing which they are beforehand) from Vault...

[FlorinAsavoaie]

@banks @bensojona should we consider consul-template as a dead project? Does Hashicorp need the community to completely take it over?

[pearkes]

@FlorinAsavoaie Sorry for responding slowly from our side. consul-template is definitely not dead -- it is heavily used by many users of Consul and Vault and we do keep an eye on issues here.

Unfortunately, many useful contributions (or questions around contributing) such as yours haven't gotten the responses we'd like given the workload from our other projects. I can imagine you understand the trade-off here.

However, I can respond to your most recent question, I think!

In order to make things more efficient (although I totally don't understand why this is needed - why would you generate the same file over and over again, on the same system), consul-template has a very interesting approach of not rendering a template more than once if the template source code has the same MD5 as another template source code.

This is part of what is needed for de-duplication mode -- so many instances of consul-template can optimize the amount of queries made to Consul by having one instance of consul-template perform the (typically many) queries to render a template, and then store the result in the Consul k/v tree. Other instances then watch that rendered template in the k/v and only render it if they keys in the original template changed. You can read more about that here.

I think your comment: https://github.com/hashicorp/consul-template/issues/1077#issuecomment-373426949 describes the use-case you want quite well, so thanks for that. I do agree that given the above it may not be possible in the way you describe it there.

[FlorinAsavoaie]

@pearkes I think that the optimization using the md5 hash over the template is in fact a really bad one. That's because, I think, in most cases, the same key is indeed used more times, but usually by different templates. As such, the md5 will differ and the queries will still be made. A much better optimization would be to have a shared brain, at least for the Consul dependencies of a template.

The optimization you are talking about is only if you use whole parts of a template in more files and define them as subtemplates on their own.

[dperetti]

Same requirement here. Trying to generate certificates from consul in a generic way led me here :-)

[pierresouchay]

For those having this issue, consul-templaterb does support writting to any file this and might be used to solve your issues as you might use ruby I/O for that (we are using such approach to cconfigure prometheus from Consul KV on our side): https://github.com/criteo/consul-templaterb

[gordonbondon]

We're using https://github.com/ekristen/consul-template-plugin-savetofile to solve this

[eikenb]

@gordonbondon .. Thanks for pointing that out. Hadn't seen that before.

[FlorinAsavoaie]

The problem with this approach is that the state of the generated files is not managed in any way. If a certificate written there disappears from Vault, for example, the file might remain. This can cause significant issues. The best thing would be for this to be implemented into consul-template itself in a proper way. It's sad to see this project almost abandoned by Hashicorp (I guess it doesn't make them money).

[pierresouchay]

@FlorinAsavoaie you are right about the state (you need a temp variable to store your outputs and being able to cleanup), but even so, cleanup would be non-trivial (for instance if consul-template is -9 killed, how to find the files created previously?)

We did this in some templates in consul-templaterb with hash of files stored as variables (so it works well most of the time), but I gave up implementing it as a feature as addressing templating engine abruptly and solving cleanup is really hard.

[eikenb]

@FlorinAsavoaie In your example is the reason you can't use separate templates for each certificate because there are an indeterminate number of certs and you just need to be able to loop through what is there? In other words, you can't use separate templates to render them individually and need to do it in aggregate.

Thanks.

[FlorinAsavoaie]

That's exactly the case.

[eikenb]

Thanks. I'm not working on this ATM, but am dealing with multiple templates for 1 service and was reviewing the submitted issues related to that. I do know at least a few others have had this issue (I've answered at least one question about this in the support channels) and so am keeping it in mind. I do think there is probably some general feature that will both address this issue and make other multi-template situations (like my current one) better as well.

[FlorinAsavoaie]

Well, meanwhile I switched jobs and I am not even considering using the Hashicorp stack at all here. The Open Source support is terrible. I both opened and followed more than a dozen of bug-fix or feature requests that keep getting postponed/delayed/whatever by Hashicorp and they don't even answer to provide help for someone else to implement this.Just look how much I basically talked alone on this issue. I could understand that they can't implement every feature request - I offered to do it myself and I literally ended up getting 0 help, both here and on Gitter.

[eikenb]

@FlorinAsavoaie,

I'm sorry to hear about your experiences with our FOSS efforts. We've been growing the teams as fast as we can, but haven't always been able to keep up with the community. This project, for example, was left a bit neglected when its author left for another job and it wasn't until I was hired a few months back that it had an active, company sponsored, maintainer again.

I've been where you are though and it can be very frustrating. And aside from any excuses I personally am sorry about your experiences with this project and hope if you need to use it again you will have a better time of it now that I'm taking care of it.

[FlorinAsavoaie]

And the same happened to Nomad? Highly doubt.

[eikenb]

I cannot say about what exactly happened with Nomad, but I do know all parts of the company have had growth pains as the products really took off fast and everyone had trouble keeping up with the community. We do try to work with and support the community as much as possible but we don't always succeed. I know the frustration and am sorry for my part in it.

[dmitrif]

Hi! I am wondering if there has been any new developments in regards to this? Thank you!

Similar situation, I have pomerium running as a job in the cluster, and I am using it to oAuth gatekeep + load balance internal company resources. As such the services can be spun up, and shouldn't require a pomerium redeploy; I am trying to template out all the SSL certs from vault into the instance, but failing due to the above.

Appreciate all hashicorp has done for FOSS, some truly great tools.

[eikenb]

Hey @dmitrif,

While this is still on the broad radar but not currently a priority. If you (and others) would like to see it made a higher priority you can add :+1:'s to the original issue (top post). When looking into what community requested enhancements to prioritize we use the :+1:'s to help gauge the interest. Thanks.

[chrisVia]

Another year has gone by without any movement on this, meanwhile we have been in the same boat as the original poster for a while now. If Hashicorp is not willing to work on this, could we at the very least know what Hashicorp recommends we do in this exact situation? Persisting various orthogonal TLS certificates from Vault into an application's filesystem doesn't sound like an uncommon or unreasonable use of consul-template - if it is I clearly need some education here.

Does Hashicorp really suggest if we have 10 sets of orthogonal TLS certificates to deliver to applications, that we write 10 separate templates, all virtually the same except for the vault path they query on and their destination file? We tried even abusing the command directive to split the rendered destination file (with all N sets of certificates) into respective files through awk, but clearly commands that themselves need to interact with the rendered destination file aren't really going to fly.

Are we totally missing the mark here on our desired usage pattern? If, at the end of the day, this is a misguided feature request that encourages stupid application design and/or poor Vault secrets structure, then I'd think this should be stated outright by the maintainers and this issue should be closed, rather than leaving it to linger for 3.5 years, no? But if this is a reasonable request and maintainers simply do not have enough interest or time for it (I'd understand that too, frankly), what do they suggest we do instead?

[pierresouchay]

https://github.com/criteo/consul-templaterb does support those kind of usages FYI

[chrisVia]

@pierresouchay Yup, I did read that project and think its an overall good implementation (like you say though, cleanup is a difficult problem). I would switch over in a heartbeat but the larger issue for us is that we are using local Vault-Agents and the Vault project defers templating to Consul-template so we are very much tied at the moment to this project.

[eikenb]

Hey @chrisVia,

Sorry for the lack of motion on this and other feature requests but I am (and have been) mostly only keeping up with the bugs and PRs on consul-template and it will likely stay that way for a while. I am working on a large refactor that should help with the maintainability in the long run, and give me more time for feature requests, but that is going to take some time.

TLDR; I'll probably need a PR for this feature to make it in. I would happily look at one if anyone has the time/motivation to work on it as I do prioritize community PRs along with bugs. If it can be implemented as a template function, that would be the easiest to merge as they are self-contained. Something similar to that savetofile plugin listed above would probably work.