search

haproxytech / dataplaneapi

HAProxy Data Plane API
https://www.haproxy.com/documentation/dataplaneapi/
Apache License 2.0
326 stars 76 forks source link

Add support for multiple configuration files #19

Open Xiol opened 5 years ago

Xiol commented 5 years ago

This is a feature request.

We use multiple configuration files to keep our HAProxy configuration layout less cluttered. We start haproxy like so:

/usr/bin/haproxy -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d/

All the files in /etc/haproxy/conf.d/ are then loaded in lexical order.

At the moment I can specify the main haproxy.cfg file for the data plane API to use, but then it doesn't know about the configuration under conf.d/. It would be nice to be able to specify configuration file locations in much the same way as the haproxy command can load multiple config files.

aiharos commented 5 years ago

Hi, thank you for the suggestion! We've added it for consideration for including into our road-maps. We'll update this issue appropriately.

dl5rcw commented 4 years ago

@aiharos ready like a good idea. I hate the piled up file in which we have multiple backends and frontends going. I just do ssl termination for some backends and just started with a new backend (dmz zone) going with tcp forwarding and letsencrypt. Next I am adding mx records and therefore mail-subdomains for postfix backends. It will grow and become scary to anyone trying to co-maintain. So I think that either conf.d or some sort of include /path/to/file would be fine. We could structure for mail, web, internal and tcp forwarding in separate files.

robertDurst commented 4 years ago

Curious if this is still being considered? Would be a lovely feature to have for the reasons stated above.

Warren-D-Johnson commented 4 years ago

Please consider adding a feature that allows you to include the contents of another file inline.... so haproxy.cfg might have "file_include some/path/more.cfg". Then we can separate rules into other files and it would be a heck of a lot cleaner. Kind of surprised that this feature doesn't exist already.

robertDurst commented 4 years ago

@aiharos I think this is something that should be considered, but quickly off the top of my head it seems like this adds some complexity which I assume is probably why it wasn't included in v1/v2:

  1. Do you now version across an entire directory or can you be more granular and version file by file?
  2. How do you account for files being added/subtracted from a directory?

As I understand things, with Haproxy static configs are only ever accounted for on launch/reload. That's simpler than here where you'd have to account for changes consistently.

f-leclere commented 3 years ago

@aiharos Please consider adding this feature that allows us to manage application cluster configuration. (1 common file and 1 conf.d file for each service to expose and define it's own rules and backends list generated through server-template). With current release dataplaneapi usage is very limited for our use-case. (nota: other limitations are server-template, srv and haproxy peers management to manage a real HA haproxy cluster)

aiharos commented 3 years ago

@f-leclere robertDurst pointed out some of the complexity inherent in delivering this type of feature. We would like to implement this too and we're working through prerequisites. While we don't have anything to share juet yet, the continued interest is reassuring that our efforts are going in the right direction.

alitvak69 commented 2 years ago

I appreciate all you doing but with my configuration split over 25 files I cannot use your API. Any idea on if this feature even on the road map? Also I wonder if you could use the same code that haproxy using to parse configuration files? My logs tell me The configuration file xyz.cfg in HAPROXY_CFGFILES is not defined. So it seems that you see those files based on the environment from haproxy and could load them.

Sorry for rant,

kellymenzel commented 2 years ago

This is a show-stopper for us as well. Our use-case is that we use Ansible and each service deploys its own load balancer configuration. Being able to create these configurations in separate files makes automation much easier so we went through the extra work to configure HAProxy to be able to support a config directory. Now I am looking at the API for dynamic configuration and it looks like I have to go back to the drawing board. Looking forward to any updates on this.

tylerauerbeck commented 1 year ago

Has there been any update on this? Is this still being considered?

crozone commented 1 year ago

I have a very similar setup to the OP. My primary, read only haproxy configuration is stored in /etc/haproxy/haproxy.cfg. I then also load /etc/haproxy/conf.d.

What I want to be able to do is create an additional configuration file called /etc/haproxy/conf.d/dataplane-backends.cfg so that dataplane's dynamic configuration changes are tightly scoped to that single additional configuration file, protecting the primary haproxy.cfg file from changes.

Unfortunately this doesn't appear to be easily solvable, since providing -c /etc/haproxy/conf.d/dataplane-backends.cfg doesn't provide dataplane visibility of the rest of the configuration and causes it to fail. Ideally, dataplane could take an additional list of configurations to provide it context, and then be given the target configuration to write back with -c like it currently has.