Auto optimize requirements. (Shrink / Repair / Fix conflicts)

[DavHau]

Problem

While maintaining a flat list of requirements is needed for tools like pip to increase reproducibility, the same rule doesn't apply for mach-nix. Since mach-nix' dependency resolution is deterministic, flat requirements do not contribute to reproducibility.

Mach-nix works best if only top-level requirements are specified, for the following reasons:

• dependency resolution is usually slower, if more constraints are given
• the risk of conflicts during resolution is higher, if more constraints are given

General reasons against manually maintaining flat requirements are:

• Updating top level requirements requires extra effort, since child requirements must be updated manually accordingly

Despite all that, users often input flat requirements to mach-nix, probably because one of those reasons:

• They carry over their requirements from an earlier pip workflow that required flat requirements
• They used a requirements exporter utility which produces flat requirements
• They are not aware of the advantages/disadvantages of flat requirement specifications
• They simply don't know that their requirements list includes siblings

---

Solution

If we were able to automatically transform a flat set of requirements into a top-level-only set of requirements, we would automatically reduce the chance of conflicts.

I have the following simple algorithm in mind: For each item inside the flat requirements list:

1. Compute dependency tree of current item
2. Remove all names from the original requirements list, which appear as siblings inside the current dep tree.

One side effect of this is, that the resolved dependency graph of the new set of requirements might contain different versions for sub dependencies compared to the original flat requirements.

Therefore this optimization should probably not be done by default. But I propose to implement it as:

• An option for mkPython
• A function of the mach-nix cmdline utility.
• ( An automatic fallback if dependency resolution fails )

[bmabsout]

Interop with existing tools is important. Not everyone has nix, so if random people try to run the application they should at least be able to use their usual workflow (if it works). If they want a very reproducible env, then they should use the nix tooling. Can we get mach-nix to output the flat list of requirements and their versions (ala pip freeze)? That way people using pip get the nearest approximation of the nix env and mach-nix should be happy as well. Maybe something like having a requirements string in mach-nix which specifies the top level packages, and a requirements.txt which is generated by mach-nix from that which is the flat version.