`flake.flakeModules` requirements

[roberth]

While we have a flakeModules option that works decently well in most cases, some requirements aren't satisfied yet, or at least not to a degree where it's easy, let alone obvious how to satisfy them all.

Requirements

Easy to dogfood

Due to how the module system config fixpoint works, fundamentally, we can't import from the option. This has been documented but it seems that we'll want to recast this into an easier, considering the intricacies introduced by the other requirements.

The publishing flake is larger than the dogfooded module

The publishing flake must have the opportunity to define things that are not part of the dogfooded module. For example, the consumer of the published module generally has no interest in copying the checks that come from its dependency. To avoid copying those checks, they have to be defined in the publishing flake, but not in the published module.

Access to the local flake module scope

Modules should be able to access values from the local flake, such as locally packaged build tools with their locked, known-to-work dependencies.

Deduplication and disabledModules

When references to the local flake must be made, we need to use module syntax that isn't conducive to deduplication and disabledModules.

• importApply does not set a module key, because it is too general for this. It can not assume that its invocations with different arguments should be deduplicated. That would be unexpected; or arbitrary at best.
• inline module pattern does not set a module key. While the module author could write one manually, we'd prefer a foolproof solution. This brings us to...

Module key for multiple flake modules

• flakeModules option sets a key, but this key is not encountered by modules that import other modules, because the dogfooding requirement (when applicable) does not allow access to the flakeModules option.

Flawed (non-)solutions

Status quo as documented

Does not implement the last requirement, "Module key for multiple flake modules"

Double mkFlake application

Lets you apply the module key before the main config is available, but "access to the local flake module scope" does not work properly.

Finding the local flake by recursing into the inputs graph

Leads to scalability issues and spooky action at a distance from deep dependencies; see Do not traverse inputs

Does not seem to solve the dogfooding requirement.

Ideas

Special support in mkFlake

• Can do things that a module can't do, e.g. setting top level specialArgs for the purpose of dogfooding.
• Can use lazy fixpoints instead of badly implemented fixpoint iteration-like constructs.
• Having a more standardized solution seems helpful - even if a solution could be composed from building blocks, it seems that it'd be a tricky composition.
• There's some overlap with https://github.com/hercules-ci/flake-parts/issues/119#issuecomment-1540218537. This could be good or bad.

A special importApply that just sets the module key to the file path

This might work fine in practice, but the key assumption must be made clear: do not use multiple times, as the key is always the same file path.

[Atry]

but "access to the local flake module scope" does not work properly.

Could you clarify if you mean inputs.self and self. config.flake is all you need, and it works well.

[roberth]

but "access to the local flake module scope" does not work properly.

Could you clarify if you mean inputs.self and self. config.flake is all you need, and it works well.

By local flake module scope, I mean the scope of the flake that publishes the module, as opposed to the module scope of the flake where it's imported. We can't express that distinction in one or two identifiers, because each of those identifiers is valid in both of the flakes. So local flake does not refer to the flake that imports the module through inputs.

I'd gladly adopt a less ambiguous term. Does "publishing flake" sound better to you? I think it's not entirely unambiguous because one may think that any file on github is "published"; it should indicate being exposed as a flake attribute. But that seems like a lot to ask from a reasonably simple term.

Without any help, the published module can only access the inner mkFlake's scope, so not the outer one. The outer flake might publish it again, but that doesn't solve it for the dogfooding case. Also note that unlike in the template you suggested, we'll want to be able to specify things in the flake that aren't part of the expose flake module. Let me add that to the list of requirements.

[Atry]

Finding the local flake by recursing into the inputs graph Leads to scalability issues and spooky action at a distance from deep dependencies; see Do not traverse inputs

Does not seem to solve the dogfooding requirement.

In fact it solves the dogfooding requirement by importing file path from the same flake. Importing file path is not possible in other solutions because they inject lexical scope as additional arguments, changing the module format.

[Atry]

Lets you apply the module key before the main config is available, but "access to the local flake module scope" does not work properly.

I saw your new definition of "access to the local flake module scope". I believe "double mkFlake" approach is indeed able to "access to the local flake module scope" "such as locally packaged build tools with their locked, known-to-work dependencies.". For example inputs.nixpkgs_22_11 is a local input. See https://github.com/hercules-ci/flake-parts/pull/156/files#diff-393b0b283c4654b3b3773e2e0ed527aa97cac8138423b26e28e656965cbd89e7R8