Environment config as Python

[ctheune]

batou is a fractal system that wants to get rid of the impedance mismatch between simplistic config formats and powerful programming. Environment config has become overly complex and cumbersome and we should turn environment configs into regular Python code in the spirit of our fractal components.

That would help adding arbitrary data in the environment and avoiding the whole "parse stuff from strings" dance. It would also allow calling to APIs to gather data into the environment, etc. etc. etc. ...

This also influences how we manage secrets, as our initial plan included to get rid of separate secret files to improve mergeability, but it feels weird if we'd be editing secrets directly within Python code.

[elikoga]

talked with @frlan about something along these lines, currently we have essentially 3 parts to get config from with the python attribute declaration being able to specify values as well

[Ma27]

That would help adding arbitrary data in the environment and avoiding the whole "parse stuff from strings" dance. It would also allow calling to APIs to gather data into the environment, etc. etc. etc. ...

I'm wondering if we're already better off using something which allows deeper structures. Whenever I've seen ad-hoc parsers so far is when you need lists or dicts.

Whatever the outcome is though, one thing I'd suggest to make sure that attributes are not "dependent" on where these are specified. Example (from a real-world deployment):

class SettingsBase(Component):
    attribute1 = Attribute(dict, ...)
    attribute2 = Attribute(split_colon_into_dict, ...)

Effectively, attribute1 can only be configured via a subclass while attribute2 can only be reasonably configured in the environment config (thouhg it would be theoretically possible to do that in a subclass as well).

As a beginner it's tempting to assume that Attribute(dict) is the way to go (and even more with Attribute(list)) and that ad-hoc parsers are problematic was already mentioned.

[ctheune]

Yeah passing a factory/type conversion to the Attribute that isn't prepared to handle strings as inputs is bad.

IMHO those are good examples why I'd like to switch to Python in general so we loose this layer of conversion.

[Ma27]

IMHO those are good examples why I'd like to switch to Python in general so we loose this layer of conversion.

I had a few thoughts about that which I'd like to share here, perhaps these are helpful in sketching out the design for that:

• I'm not sure what your proposal exactly looks like, but with plain inheritance we may be able to get autocompletion (and maybe even typechecking) for free with tooling like python-lsp-server.
• When using Python for that, we'd enable people to put logic into the (secret) configuration files which may be used to e.g. configure a service in the environment config when it's only needed on production. Right now you'd have to work with feature flags (not sure if that's the correct term, I mean the component:feature syntax) or attributes and I'm not sure if putting logic into the configuration is desirable in the context of batou. What I'm wondering is, do you intend to use components for that or a different "format"? Especially in the former case I think it might be good to think about how much logic is OK in (environment-specific) configuration and document that explicitly.
• In an AppOps meeting a while ago we discussed about "mergeable" secret files, i.e. a format that's trivial to git cherry-pick when changing a single secret. It was suggested use a single symmetric key and encrypt that one for all recipients, i.e. every member of the RG for instance to reduce the diff when adding/removing people. This is basically what sops is doing for numerous keys including age & pgp. AFAIK that's only doable with YAML as data structure when using sops, so we'd need to come up with our own implementation for that problem. Probably not that of a big deal, but I still worth mentioning IMHO.

[ctheune]

Integration with Type Annotations would be really helpful. Yes. batou hasn't caught up on that end at all.

The last item is currently being worked on to provide diffable age files. We're sticking with the ini format there but encrypt all the values. Whether to roll another layer of encryption ourselves to reduce the noise is a good question. We'll need to look at this within the context of #440. Could make sense, but "don't roll your own crypto" and such ... OTOH the primitives are all there already ...

Python to configure environments would not mean that we end up with encrypted Python files ... ;)

[Ma27]

Python to configure environments would not mean that we end up with encrypted Python files ... ;)

Sure!

My point was that I'm not sure if we could achieve that in case we want to integrate sops which would also comply with "don't roll your own crypto". But again, that's an idea I had and I'm not sure if we even want that, but I wanted to remind you of that potential consequence here :)