kelseyhightower / confd

Manage local application configuration files using templates and data from etcd or consul
MIT License
8.35k stars 1.41k forks source link

Create multiple config files from templates #256

Open lorenz opened 9 years ago

lorenz commented 9 years ago

I want to run tinc with confd, tinc requires you to have a configuration file for every host. These can all be built from the same config. I'd like to have the host list and the host-specific configuration stored in etcd and afterwards turned into files by confd. Any chance that this is getting added?

kelseyhightower commented 9 years ago

@lorenz What would that look like? What data would each template receive?

lorenz commented 9 years ago

@kelseyhightower Basically there would be a go template which is the same for every file. The data would be equal (structure-wise, not data-wise) but grouped up in directories, one for every file. Does that make sense?

mborho commented 9 years ago

+1

The possibility to generate config files programmatically would be very handy.

kelseyhightower commented 9 years ago

@mborho Can you give me an idea of what an example resource template and template file would look like?

mborho commented 9 years ago

@kelseyhightower Good question :) Let me start with the use case first. Having some keys in the following way:

/auth/foo/user1
/auth/foo/user2
/auth/bla/user1

Which should result in multiple files after processing:

foo.htpasswd
bla.htpasswd

Currently I'm using an intermediate file, which gets cut by some awk magic added to the _restartcmd.

But perhaps a custom tag is possible in the templates, something like this:

{{configfile "filename" "fileprefix" "content"}}

And in the template resource something like:

dest_dir_multiple = /path/for/configfiles/

So the config-files would be saved as /path/for/configfiles/[filename].[fileprefix]

Makes this any sense for you?

crunchywelch commented 9 years ago

FWIW, this maybe addresses this issue: https://github.com/kelseyhightower/confd/pull/332

kelseyhightower commented 8 years ago

After thinking about this for a while, I'm going to reject this feature in order to keep confd simple. confd is really designed to map one template resource to a single output file. This keeps confd simple, but as you pointed out it does limit the types of problems confd can solve.

I'm willing to make this trade off though in order to keep confd simple.

openxxs commented 8 years ago

Hi, @bacongobbler I need to confd a nginx.conf file and a ./conf.d/upstream.conf file, which nginx.conf includes ./conf.d/upstream.conf. Because these two files are determined by different etcd clusters, I have to start a confd process for each. toml file for nginx.conf:

[template]
prefix = "/domeos/loadbalancer/nginx/nginx03"
src = "nginx.conf.tmpl"
dest = "/etc/nginx/nginx.conf"
keys = [
    "/conf",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/nginx -s reload"

toml file for ./conf.d/upstream.conf:

[template]
src = "upstream.conf.tmpl"
dest = "/etc/nginx/conf.d/upstream.conf"
keys = [
    "/registry/pods/default",
]
check_cmd = "/usr/sbin/nginx -t -c /etc/nginx/nginx.conf"
reload_cmd = "/usr/sbin/nginx -s reload"

When I change the /registry/pods/default value and generate a fault ./conf.d/upstream.conf, confd tells me:

INFO /etc/nginx/conf.d/servers.conf has md5sum 457c221119f29b5eeaa0a45ddae63d52 should be 46d8305693d13c7fbaf8d509bb284c94 2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: INFO Target config /etc/nginx/conf.d/upstream.conf out of sync 2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: ERROR "2016/06/28 02:40:19 [emerg] 436#436: \"upstream\" directive is not allowed here in /etc/nginx/conf.d/upstream.conf:4\nnginx: [emerg] \"upstream\" directive is not allowed here in /etc/nginx/conf.d/upstream.conf:4\n" 2016-06-28T02:40:19Z 26c17fae16ff ./confd[426]: ERROR exit status 1

It means check_cmd works. But the ./conf.d/upstream.conf has been updated, which makes nginx product ERROR. Dose check_cmd have to be /usr/sbin/nginx -t -c {{ .src }}? What if check_cmd = "bash myCmd.sh"?

bacongobbler commented 8 years ago

check_cmd can be whatever you want it to be. It could literally be /bin/true if you don't care about validating your templates.

waszi commented 8 years ago

I am using confd to configure nginx on multiple containeres, domain config/maps etc it is easy to split config without multiple destination but there is small exception. SSL Certificates requires separate files with name based on domain name. Any ideas how to solve this ?

ghost commented 6 years ago

Hi @kelseyhightower , do we support multiple destinations for config files now?

okushchenko commented 6 years ago

@ricky3dec there is no support for multiple destinations yet. Can you share your use case?

godefroi commented 6 years ago

I have a use case. I use confd to write out rules files for Prometheus. I want each group in a separate .yaml file, and I have each group in a separate key in Consul. I would like to use a single resource to write each key into a separate file, since I don't know what rules groups (keys) will exist ahead of time. It might look like this:

[template]
src = "prometheus_rule.yaml.tmpl"
dest = "/prometheus/rules/{{$key}}.yaml"
keys = [
        "/prometheus/rules/*",
]
okushchenko commented 6 years ago

@godefroi I think that this way of maintaining templates is convenient for an operator, but only for manual management. If you are using configuration management tool, like Confd, there should be no justification to maintain that approach of separate config files.

Will a single config file with a loop inside work for your use case?

godefroi commented 6 years ago

Yes, and that's ultimately what I did. It may not be possible in the general case, however.

okushchenko commented 6 years ago

@godefroi I'll reopen this issue for now. I think this issue needs more discussion. Let's wait for more use-cases to be shared by a community.

fiatflux commented 6 years ago

Wireguard also requires a separate file per device. I would like to use confd to generate these configuration files so that I can dynamically add and remove tunnels in a VPN mesh. The workaround of breaking up one generated file into multiple files isn't hard, but feels a bit clunky.

hubo1016 commented 6 years ago

It could be implemented with some hack:

  1. introduce a new option multi_dest or variable_dest
  2. introduce a new template function setdestination(path, prefix='/')
  3. generate an UUID (or some random hash) to create a boundary
  4. when setdestination is called, output something like --<UUID>--dest=/newpath--<EOL> to separate the output file
  5. split the file and copy each part to different file
  6. run check_cmd and reload_cmd

The check_cmd and reload_cmd should support more parameters in this mode.

okushchenko commented 6 years ago

@hubo1016 it's a hack. I've tried to find a simple solution to this problem, but I just can't find one. If someone will propose a decent solution I'll gladly consider it.

pantierra commented 6 years ago

Thanks a lot for this nice piece of Software!

My two cents, with a use case: A multi-domain prosody (jabber server) I want to run through docker and to generate its configuration on the fly based on a list of urls. prosody creates vhosts - similar to a web server - ideally with different files for each domain. confd with the here requested feature would be perfect for this. I guess loops are enough for me, it is just not that elegant.

metanovii commented 6 years ago

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

okushchenko commented 6 years ago

@xamanu @metanovii thanks for sharing your use-cases!

jinxmcg commented 6 years ago

I also need this for generating multiple interface files at setup. The workaround I did was to use confd to generate a .sh script (parameters functions etc) which is executed with "reload_comd" and generates the multiple config files I need.

sdwire commented 5 years ago

My use case: provisioning dashboards for grafana from a central source. From what I can tell in the grafana docs, each provisioned dashboard must be a separate JSON file in, for example, /var/lib/grafana/dashboards. I'd like to use confd to pull those dashboards from consul's kv store (where all the rest of our config lives) and create one JSON file for each key in a consul path.

ykfq commented 5 years ago

Its really useful to seperate the dest files as our nginx configuration file is growing very large with thousands of lines.

poppyred commented 4 years ago

+1

Yullin commented 4 years ago

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

so, about this case, any one had best practise case?

metanovii commented 4 years ago

Hi, my case: auto generate certs for some domains. Without this func i must create template for each domain.

so, about this case, any one had best practise case?

hi, now i use consul-template. One template for all certificates.

tigerfsh commented 3 years ago

@metanovii Can you share the link how you use consul-template?

metanovii commented 3 years ago

@metanovii Can you share the link how you use consul-template?

Now i use hashicorp vault + consul-template. Some cronjob generate certificates and put into vault and consul-template get certificate from vault.

example config for consul-template:

reload_signal = "SIGHUP"
kill_signal = "SIGINT"
max_stale = "10m"
log_level = "warn"
pid_file = "/var/run/consul-template.pid"

wait {
    min = "5s"
    max = "10s"
}

vault {
  address = " "
  token=" "
  grace = "10m"
  unwrap_token = false
  renew_token = true
}
syslog {
  enabled = true
  facility = "LOCAL5"
}

template {
  contents="{{ with secret \"kv/somepath/infra/security/x.509/somedomain\" }}{{ .Data.data.crt }}{{ end }}"
  destination="/etc/nginx/certs/somedomain.crt"
  perms = 0600
}
template {
  contents="{{ with secret \"kv/somepath/infra/security/x.509/somedomain\" }}{{ .Data.data.key }}{{ end }}"
  destination="/etc/nginx/certs/somedomain.key"
  perms = 0600
}

.....

exec {
  command = "systemctl reload nginx"
}
wurenny commented 3 years ago

multiple dests are very useful, e.g: i want add some nginx proxy servers to nginx conf.d by rest api, i can do this:

maybe it is not sure how many proxy servers, and dest file of confd template must be dynamic