Improve maintainability of hydra build toolchain

[locallycompact]

Why

The Hydra (protocol) developers need a convenient way to build and update dependencies of the project. This also includes provisioning of development tools in the right version (aka dev shell), static builds, and docker images.

Furthermore developers wanting to use packages, should be able to easily depend our libraries and get started quickly.

While our primary way to use the project should remain using nix, we want to keep support for a non-nix way to build and develop using cabal only (with some caveats on getting the right versions of tools).

What

• Dynamic and static binaries (for linux), as well as docker images can (still) be built

• Not need to create a build plan from scratch when initially building the project.

  • nix build does not use a solver to find a build plan
  • cabal build effectively just uses a pre-selected set of packages as well (could be trivial constraints - already solved)

• Updating dependencies workflow is changed

• Downstream projects can depend on our libraries

  • use a flake input to get our packages (nix users)
  • copy our cabal.project constraints (cabal users)

• Should be able to update the build plan for all users

  • by updating the nix package set (nix users)
  • TBD: How to do this for cabal-only users (who do not have nix installed)

• The nix flake outputs are fully inspectable for all / on all platforms. i.e. nix flake show and nix flake check just work

How

• Avoiding IFD by using horizon instead of haskell.nix (which calls cabal in an IFD)

• TBD: Persist horizon package set into cabal.project as a set of constraints (or use cabal freeze / lock files?)

• TBD: cross-toolchain-compilation with muslc still needs to be possible

• TBD: hoogle with all dependencies included in the index in the dev shell

• TBD: a working HLS (currently patched, any latest is fine)

• TBD: check whether all haskell.nix modules are still doable with horizon

---

Detailed rationale

Our nix setup is causing a lot of complications both for us and for downstream consumers of hydra libraries. The most serious of these is:

The presence of IFD when constructing the dependency graph means the flake can not be evaluated statically. Consequently it is very hard to debug attribute values when an evaluation fails, particularly when debugging the expression for a non-native system.

There are also several quality of life issues that I believe should also be addressed.

1) Implicitly chosen dependencies mean that downstream consumers of hydra have to solve build plans themselves in addition to actually developing their application. Since we have solved this build plan ourselves, we should be able to share this result with them as an input value to their project directly, but we can't. Since IFD will recompute the build plan each time, any flake update the user makes can cause build issues that they themselves can't debug.

2) We also have the same problem when trying to resolve compatible versions of our own dependencies, that others upstream have already partially solved. We essentially redo this work every time.

3) We can not derive our CI directly from the flake, as we can not evaluate it statically. With a static flake, we could turn the flake into CI jobs purely, as well as including linting checks like fourmolu.

4) We can not detect breakages to reverse dependencies automatically. For example, if a bump to cardano-api were to break hydra.

I propose Horizon Haskell Stable Package Sets as a solution to these issues. Horizon Haskell uses explicitly pre-computed stable package sets as the input to a build plan that can be evaluated statically without IFD. We can use this package set as an input both to our own nix code, and offer it to consumers of our packages containing a collection of libraries that all work together. We would also have a package set repository (horizon-hydra) that could receive pull requests, creating a common place for all consumers to contribute to advancing the build plan forward, sharing the work of finding working build plans and reifying them explicitly as a common input.

Horizon stable package sets are also maintainable by non-nix users with testimonials to support that claim. (They will be on the website soon.)

Drawbacks: Some of the features of haskell.nix are not yet supported by horizon, such as static builds and windows cross compilation. Static builds are likely not going to be an issue, and I would potentially suggest not solving windows XC with nix at all. We could compile horizon's data model directly to a cabal.project and compile natively.

There is an example branch here, with instructions. It is incomplete, but the remaining work is just more of the same - as well as initializing a new horizon-hydra repository as an extension to horizon-plutus.

https://github.com/input-output-hk/hydra/commits/lc/horizon-example

[ch1bo]

Discussed purpose of this in the grooming:

• Increase "productivity" in the team (hope)
  • Dan is confident that maintainability is better (especially as he could help more)
• Reduce frustration with nix in the team (hope)
• Making it simpler to depend on hydra Haskell packages (is it?)
• Initial set up time for developers shorter / less to download?

We could interview our (more external) developers what would help them in using our stuff?

• @cardenaso11 what do you think?
• @uhbif19 any opinions?
• @yasuke you may have a strong opinion (and own workflows at obsidian) - anything which would be more compatible with that workflow?

---

The above could / should be distilled further into this item (updating the description) before we do fully buy into this.

[uhbif19]

@ch1bo

Ability to build without Nix is definitely cool. While, AFAIU separating of public api package (#722 ?) and putting it on CHaP are intended way to bring that ability for dev users of Hydra.

Also I do not know what is the best way to import Hydra with Nix, some copying of Nix hacks is done in our flake, which is probably not considered good. AFAIU it is not documented now, it would be cool if you would place your recommendations on that.

As for non-dev users of Hydra, I do not know if static binaries are important, cuz we always need to use Docker anyway.

FYI @GeorgeFlerovsky

[cardenaso11]

@ch1bo

Very much support! Hit a bunch of issues getting team up to speed with Hydra development previously, back before arm64-darwin support was added. While I don't think that target will be an issue again, it's very hard to convince people to use Nix in my experience, regardless of whether or not it works for a platform.

Not having static builds without nix is fine in my opinion, but static builds (with or without nix) are pretty important for my current work, I'd prefer they're supported in some kind of way before merging any switch

[GeorgeFlerovsky]

I believe that binaries are important and I strongly support ensuring that alternatives to nix are available to people. Nix is good for some things, but it's a huge maintenance/adaptability burden to others.

[Yasuke]

The nix related improvements sound promising, and I would be excited to see those be done.

For the common case of needing hydra binaries it might be enough to just have the toolchain produce assets for more architectures and include those in each release as well as docker containers, so that building from source isn't a requirement for most. Alleviating a lot of the nix usage by end consumers in the process.

For those building from source who only need the binaries, a non-nix build is feasible requiring a machine that has:

• GHC
• Cabal
• Libraries
• Packages and then you can build, the problem is you may also want/have other versions of those things at the same time on the same machine, or you may have different versions across machines, or between deployments. We use nix because it allows us to have these dependencies for each piece of software or each version specified in a way that any developer or any machine can make use of it.

For developers consuming hydra libraries in a project: The complexity problem is the project they are working on will likely have different GHC and package requirements than Hydra itself. In a lot of cases the project requirements aren't malleable.

This means that developers will have to either:

• Adopt the versions of tools and packages that Hydra uses, which isn't always feasible (for example on Hydra Now we need a specific version of libraries like time and aeson, as well as a specific GHC to facilitate cross compilation to javascript and android).
• Find a common package set and GHC version that works for the Hydra libraries' and their own project's needs. This is always going to be difficult, especially when different areas of the cardano ecosystem require fairly different versions of common cardano packages. Potentially finding and setting appropriate version bounds in the cabal file and testing with many compilers would likely help a lot, but would be a large undertaking, and in the worst case developers still much patch Hydra and/or do the dependency solving themselves.