Develop replacement for `ConfigManager`

[mih]

This is a key component of any datalad code. It is present in any datalad session datalad.cfg and part of any Dataset or *Repo instance.

The implementation in datalad-core is old, has various spaghetti pieces, and comprises at least 3 different APIs.

I believe this code needs an update for a number of reasons:

• complex logic that is hard to understand from the given implementation
• origin information of config items is complicated to obtain (both internally and from a user perspective), but for security reasons it is often desirable to be able to ignore sources that can be manipulated "externally" (e.g. via a "pull")
• the scope setup does not match those of git-config -- ie. no support for worktree
• the notion of overrides is clashing with Git's own command scope -- see #325 for the struggle to sort this out

Given the essential nature of this code, I think a replacement must

• be initially API compatible with the present ConfigManager
• shrink the API to what is needed (not what is possible; right now it is dict, and ConfigParser in Python and ConfigParser in GitPython, and some other pieces) via deprecations resulting actual removal
• be better aligned with git-config in terms of scopes and concepts
• be as fast or faster
• come with minimal dependencies

TODO:

• [ ] Assess actual API usage
• [ ] Look for code duplication with other datalad code, and (re)use of (internal) helpers (e.g., for parsing .gitmodules) to inform refactoring
• [ ] (Re)evaluate principle behavior (right now we read all config at once, and write individual items more-or-less one-by-one, but other behaviors are possible)
• [ ] Propose design
  • better separation of read and write?
  • type support?
  • recognize scopes distinct from origins of config items
  • optimal internal/runtime representation of configuration items? presently optimized for repeated reads within a Python session, with reloads and writes being expensive
  • double-check API to support all desirable operations (e.g. something like replace-all within a particular scope...)
  • no longer rely on generic CommandError to communicate permission issues when setting (e.g. system) configuration
• [ ] Implement and benchmark

Thoughts

Maybe have a dataclass

GitConfigItem:
   scope: GitConfigScopeType
   origin: str
   key: str
   value: str | Any

maybe with additional accessors like

• section
• name
• asbool()
• asint()

The internal representation within ConfigManager could be a list of these items, with amend/overwrite/replace rules applied on access. Or a series of lists, grouped by scope in a dict (likely scope removed from GitConfigScopeType then), or even grouped by (scope, origin) to facilitate reloads.

It could also make sense to have an explicit GitConfigMultiValueItem, that exposes a get_all() equivalent, rather than (only) having ConfigManager.get(getall: bool)

[adswa]

linking a few config manager related issues from core:

• https://github.com/datalad/datalad/issues/6551
• https://github.com/datalad/datalad/issues/5155
• https://github.com/datalad/datalad/issues/4179

[mih]

More corner/use cases are:

• be able to turn on/off include directives
• knowing which values (of multi-value items) came from which source
• honor "protected configuration" different than other (Git treats these scopes as if they are controlled by the user or a trusted administrator. This is because an attacker who controls these scopes can do substantial harm without using Git, so it is assumed that the user’s environment protects these scopes against attackers.
• honor GIT_CONFIG_NOSYSTEM
• be aware of GIT_CONFIG[_GLOBAL|_SYSTEM]

[mih]

The ping in https://github.com/datalad/datalad/issues/5383 reminded me that the present config manager also has no support for git annex config. And this one is special, because it has a fixed set it items, and rules of "engagement" are more complex

[mih]

Making a start for this effort now.

[mih]

https://github.com/datalad/datalad-core/pull/19 provides a new ConfigManager implementation, built to support all aspects gathered in this issue. Even though, it may not actually support all proposed features at this point.

https://github.com/datalad/datalad-next/pull/760 has an exploration whether it would be possible to implement an adapter for the new manager with the API of the old one. This seems possible, except for some edge cases (some of which a more like bugs than features).

I am closing this issue here. Feature requests can be posted to https://github.com/datalad/datalad-core