Feature Request: TOMLDocument deep merging

[howroyd]

It would be a very handy tool to be able to easily deep merge TOML Documents.

As a use case which I am currently facing, and having to roll my own recursive merger...

Main config.toml (which probably gets checked in to source control):

[server]
hello = "world"
[server.config]
url = "1.2.3.4"
username = "defaultuser"

And a non checked in secrets.toml:

[server.config]
username = "adminuser"
password = "qwerty"

Then I would like to:

with open('config.toml', 'r') as f:
        config = tomlkit.load(f)
with open('secrets.toml', 'r') as f:
        secrets= tomlkit.load(f)

config.update(secrets, override=True)

With the expected output being equivalent to:

[server]
hello = "world"
[server.config]
url = "1.2.3.4"
username = "adminuser"
password = "qwerty"

[frostming]

We are not going to expose such helpers, you can write your own instead.

[howroyd]

We are not going to expose such helpers, you can write your own instead.

Would you be interested in it as a PR @frostming ?

[frostming]

We are not going to expose such helpers, you can write your own instead.

Would you be interested in it as a PR @frostming ?

Not either, this library only exposes TOML document that can be operated interactively with python types(I know that is not perfect, but I hope so). Not any fancy APIs on that. One question: does anyone want such helpers with Python dictionary? And why doesn't CPython provide that?

[howroyd]

Nothing's perfect but this is a very good and helpful library, which is why I was keen to offer support :) Thanks for your efforts!

Python3.9 provides the pipe operator | for merging a dict with override of existing keys, and this works with tomlkit but it isn't deep, so you lose deep keys that aren't in the second (in this case, server['config']['url'] vanishes.)

Is some thrown together code is helpful, trying to stick to my original use case:

import tomlkit

main_config = """[server]
hello = "world"
[server.config]
url = "1.2.3.4"  # <- note this entry
username = "defaultuser"
"""

secret_config = """[server.config]
username = "adminuser"
password = "qwerty"
"""

main = tomlkit.parse(main_config)
secret = tomlkit.parse(secret_config)

my_config = main | secret

print(my_config)
# {'server': {'config': {'username': 'adminuser', 'password': 'qwerty'}}} # <- note missing "url" entry from main_config because server.config was not deeply merged.  If we don't merge in the secret config (e.g. non admin user) then we get the url entry ok.

And a thrown together working example of the end goal, showing the problem) is here: https://godbolt.org/z/95bs5cEGG

You are right that this is a general Python Mapping type issue, however, rather than tomlkit specific. This is just where I came across the issue. I am surprised it isn't in itertools or similar.

Hope that answers your question! Cheers