barthisrael commented 10 months ago

Description

This PR adds support for config models in Barman. The main idea of this feature is to make it easier to integrate Barman with HA solutions, e.g. Patroni.

Besides the usual Barman server configuration, we now add support for Barman model configuration. The model acts as a set of overrides for configuration options for a given Barman server.

The servers configuration now have an additional configuration option:

cluster: it defines the cluster name for a given Barman server. If not set, defaults to the server name -- for backward compatibility.

In order to say to Barman that a section of the configuration file is a model, you need to set the configuration option model = true in such section. Also, as explained for the Barman servers, you need to specify a cluster option in the model. That is what makes it possible to apply config models to servers: they need to belong to the same cluster.

The models currently support overriding all server configuration options, except for:

Path related options:
- backup_directory
- basebackups_directory
- errors_directory
- incoming_wals_directory
- streaming_wals_directory
- wals_directory
Hook related options:
- post_archive_retry_script
- post_archive_script
- post_backup_retry_script
- post_backup_script
- post_delete_script
- post_delete_retry_script
- post_recovery_retry_script
- post_recovery_script
- post_wal_delete_script
- post_wal_delete_retry_script
- pre_archive_retry_script
- pre_archive_script
- pre_backup_retry_script
- pre_backup_script
- pre_delete_script
- pre_delete_retry_script
- pre_recovery_retry_script
- pre_recovery_script
- pre_wal_delete_script
- pre_wal_delete_retry_script

Note: we did it that way because those options can be easily destructive for a Barman server, with the most concerning ones being the directories related as they specify where to store files.

Example

As an example, let's assume you have a PostgreSQL cluster composed of 3 nodes, say A (primary), B (standby), and C (standby), and assume you are taking backups from A through a Barman server named my_cluster. You could create a couple config models which would override the conninfo options of that Barman server. So, in the event of a failover in the cluster, you could ask Barman to apply a given model on top of the Barman server config, overriding the values that would reference the failed node.

Please note that you can either have no active model, or have at most one active model at a given point in time. If you have no active model, only the options from the Barman server take place. If you have an active model, the options defined in the model will override the corresponding options from the Barman server config.

In order to switch configuration models, the user can use the new CLI command barman config-switch. You need to call the command specifying the server name and the model name. That command can also be used with the --reset flag, so the active model is reset, allowing only the server options to take place.

Technical details

Previous to this PR we used to have a ServerConfig class in the code. It used to have the logic for handling the configuration of the Barman servers. While introducing the models implementation we noticed some overlap between how we would handle server and model configuration. With that in mind we created a BaseConfig class which handles the similarities, and created a couple subclasses from it:

ServerConfig: the usual ServerConfig which existed before, with an additional cluster option being supported. Besides that, it now also tracks a few things related with the server models:
- The hidden file .active-model.auto, which is written under the Barman server directory when a model is active for that server, if any;
- It also exposes a method apply_model so both Barman and the user can request a given model to be applied
- to_json method has been changed, so it correctly points the source file for an option that is override by a model
ModelConfig: a new class to handle configuration of Barman models. It exposes a few handy methods:
- get_override_options: yields the options and values which are set by the model, and which override the Barman server config;
- to_json: so we can put models information in the diagnose output

We had to change the logic for parsing the Barman configuration, so it is now able to distinguish between sections for Barman servers and for Barman models. The same function which was used to parse and cache the servers from config files is not in charge of parsing and caching the config models. At the very end it will apply any configuration models which are active based on the hidden files that are maintained by Barman inside each server directory.

When parsing config models, Barman will log warning messages about options which are not supported in models. Also, it will deactivate Barman servers if they attempt to apply an invalid model.

References: BAR-122.

barthisrael commented 10 months ago

Thanks for the new review! I've applied most of your suggestions -- except for the one which depends on the order of merging between this PR and #868 .

I'll now work on adjusting the oversight in regards to cluster vs servers relationship -- it should be 1 to many, not 1 to 1.

barthisrael commented 10 months ago

@gcalacoci I just submitted a commit which changes the behavior of ModelConfig, so it allows overrides based on a blacklist instead of a whitelist on the server config options.