marzer / tomlplusplus

Header-only TOML config file parser and serializer for C++17.
https://marzer.github.io/tomlplusplus/
MIT License
1.53k stars 146 forks source link

Recursive merge #155

Closed mark-99 closed 2 years ago

mark-99 commented 2 years ago

I want to implement a config system (using TOML) which supports inherits/includes type functionality, so I can have a base config and specializations, or include standard blocks in the final config (in order to avoid the usual DRY/copy-paste problems that configs inevitably end up with).

To be clear, I am not wanting this as some first-class feature in TOML, just something I can implement on top. I also realise this is not the same thing as appending or textually including .toml files.

It seems what is needed is a table merge. Something like C++17's std::map::merge(), or std::merge(). https://en.cppreference.com/w/cpp/container/map/merge

Although ideally I'd like to merge into a subtree (to include standard blocks in different sub-keys).

Either: (a) Some guidance on how to implement this using the existing public API. I had a go using iteration, visit and insert but couldn't get it working. and/or: (b) The functionality added to the API, e.g.

Could either do inplace merge where one table is also the output, or return a new table. Optionally add overloads where the whole table is passed to cut down on .begin() noise.

void toml::merge(iterator dest, iterator src, toml::replacement_policy = replace); // or src,dest depending on preferred style Where replacement_policy is e.g. keep, replace, throw. And/or: tbl toml::merge(input1, input2, replacement_policy);

Pseudo-code:

//////////////////////////////

auto base = load("base.toml");
auto overrides = load("overrides.toml");
auto config = toml::merge(base, overrides);

base.toml;
[section1]
key1="value1"
key2="value2"

overrides.toml:
[section1]
key2="value3"

Output:
[section1]
key1="value1"
key2="value3"

//////////////////////////////

auto table = load("my_server.toml"));
if (tbl.contains["inherits"])
    toml::merge(table, load(tbl["inherits"]), replacement_policy::keep);

server_base.toml:
[server]
ip=192.168.100.100
debug_mode=false

my_server.toml:
inherits="server_base.toml"
[server]
debug_mode=true

Result:
[server]
ip=192.168.100.100
debug_mode=true

//////////////////////////////

// Put a subtree into 2 different places in the top-level table.
toml::merge(find("client1"), load(tbl["default_client_config"]), replacement::policy::throw);
toml::merge(find("client2"), load(tbl["default_client_config"]), replacement::policy::throw);

default_client_config.toml:
connect=true
print_thing=false

config.toml
[server]
ip=192.168.100.100
debug_mode=false

[client1]
include="default_client_config.toml"

[client2]
include="default_client_config.toml"

Result:
[server]
ip=192.168.100.100
debug_mode=false

[client1]
connect=true
print_thing=false

[client2]
connect=true
print_thing=false
marzer commented 2 years ago

Hi there, thanks for the suggestion! I won't be adding this to the public API, mostly because I don't want to lock in a merging behaviour and then later find the TOML standard has changed to issue some sort of decree on how merges work.

Fortunately it's entirely achievable with the existing API; I've added the toml_merger example to demonstrate one way of going about this (it implements your first option). Should be fairly straightforward to extend to do the "inherits" bit, that's left as an exercise to the reader :)

Feel free to ask for more info here if necessary, or on gitter: https://gitter.im/marzer/tomlplusplus