Actually build new (and existing) packages in CI

[michaelpj]

The gold standard for CI for CHaP would be to check when adding a new package foo-X:

1. That foo-X builds from CHaP
   • If this fails the package is probably broken and shouldn't be released (might need bounds!)
2. That all the other packages in CHaP still build (foo-X could supplant foo-(X-1) in build plans, causing failures)
   • If this happens that tells us that the other packages need to be revised to add upper bounds, which is great and we can do it in the same PR!
   • This is only remotely feasible because we have a small repository, but I think we could do it!

Doing this is going to be tricky:

• We need a hackage.haskell.org index-state to use. We might just have to have a cabal.project for the CHaP CI and update the index-state occasionally.
• We need to work out how to build them, including all their system deps and so on. We probably need to use haskell.nix for this. We will then need to record any special Nix setup (e.g. the forked libsodium) somewhere. Probably doable, but needs some thought. Ideally this will get lots of cache hits so the CI will be fast!
  • If we ever want to build Haddock we're going to need this anyway!

For bonus points, if we have this setup, then it's also something that can be used locally to test whether a new addition works properly.

[andreabedini]

We need a hackage.haskell.org index-state to use. We might just have to have a cabal.project for the CHaP CI and update the index-state occasionally.

My thinking would be to use the same index-state as the timestamp in consideration on chap. I know it's possible to select different index-states for different repos but I tend to think we should get away from that (i.e. cabal update will update all repos if you don't specify anything)

[michaelpj]

My thinking would be to use the same index-state as the timestamp in consideration on chap.

Ah, you mean the timestamp of the package that's getting added? Or the latest one. Yes, that would work!

[michaelpj]

Proposed plan:

1. Build the newly added package
2. Build a fixed set of additional packages as a smoke test (e.g. recent versions of cardano-api)
3. Build everything.

I think 1&2 are pretty doable and 3 is hard but probably can be left for now.

[andreabedini]

I am trying to gather the experience of the past two weeks of troubleshooting build plans into a sort of workflow.

I am using a modified cabal that does the equivalent of adding extra-packages in an otherwise empty cabal.project and building those packages (being also able to specify constraints). Then I am using a file+noindex repository where I collect cabal file revisions.

This setup allows me to check that every package version has a working build plan, and optionally to fix it. Cabal new-style builds are quite effective at caching so that works too.

This is basically a step towards 3 but I am happy to build only the most recently added package (or perhaps it's the effectively the same thing given the caching?).

One question I have is, what if nothing works? :-) almost every package needs a revision (which we are holding back to not break haskell.nix).

[michaelpj]

One question I have is, what if nothing works? :-) almost every package needs a revision (which we are holding back to not break haskell.nix).

Right, so our current state is going to be disastrous, and almost nothing will build in isolation (well, I think e.g. the plutus packages might be fine!). But I think it's still valuable to have the check in place, even if it's not mandatory for it to pass. It's only really once we get revisions and/or people start putting better bounds in their cabal files that it'll be green mostly.

I am using a modified cabal that does the equivalent of adding extra-packages in an otherwise empty cabal.project and building those packages (being also able to specify constraints). Then I am using a file+noindex repository where I collect cabal file revisions.

For the CI just building the packages that are being added, I wonder if we can do something simpler using haskell.nix: just call hackage2nix on the built index, and then build the package from there (with some fixed configuration that we're going to need for e.g. the libsodium fork). Obviously this mostly won't work right now until we get revisions, but it would be a nice start?

[andreabedini]

But I think it's still valuable to have the check in place, even if it's not mandatory for it to pass

:+1: and turn it mandatory when we have revisions working?

I wonder if we can do something simpler using haskell.nix

Probably yes, when I looked I could not find a way to pass extra-packages to a haskell.nix project. Maybe I overlooked something obvious.

[michaelpj]

and turn it mandatory when we have revisions working?

Exactly

Probably yes, when I looked I could not find a way to pass extra-packages to a haskell.nix project. Maybe I overlooked something obvious.

So, haskell.nix has functions for building any hackage package from hackage.nix. e.g. https://github.com/input-output-hk/haskell.nix/blob/d54e2821c69b830cfa03dc47d18891797647b6da/overlays/haskell.nix#L499 Surely we should be able to reproduce this with our repository? That's all I've got so far (maybe @hamishmack has an idea?).

[andreabedini]

I have started some work in the builder branch. See example of building a package here.

[michaelpj]

I tried an approach using Nix rather than plain cabal here: https://github.com/input-output-hk/cardano-haskell-packages/pull/123

[andreabedini]

We now have a preliminary hydra job that builds all packages on CHaP. https://ci.zw3rk.com/jobset/cardano-haskell-packages/all-packages

Things still missing

• given the builder is nix a flake, its inputs as locked. Locally I'd use the --update-input flag to "unlock" the chap input (and therefore always use the latest one) but I am not sure how to do that with hydra (of if at all possible).
• we would need to integrate this into github actions.

[michaelpj]

Can we programmatically create Hydra jobsets? In some ways what we want is to drive it from GHA: build the repo, update the flake input; and then say "make me a hydra jobset which will report back to this PR" :thinking:

Alternatively, if we can get sufficient remote builders set up then I think we could just do the build from GHA. I have a request open to get access to nixbuild.net which we're now supposed to be able to use. Just hooking up to that might do the job.

[andreabedini]

We can do that, that's how @angerman github-hydra-brige works. But the problem is that CHaP is built outside nix so we cannot "re-import" it as an imput.

tl;dr I need a way to run foliage inside nix, which would need the hashes of the git repos and a way for foliage to not download them. But the way I made foliage is a bit too much like cabal (it does network I/O quite liberally). So maybe we would need a similar but different tool. Idk.

But in the short term: if we get the caching right there should no problem running nix build --override-input CHaP _repo. I can make an aggregate package with all the package components and that would make easy to build the right package.

[michaelpj]

tl;dr I need a way to run foliage inside nix, which would need the hashes of the git repos and a way for foliage to not download them.

Yes, in some ways that would be ideal, it would simplify things a huge amount. But I think it would be a lot of work at best.

But in the short term: if we get the caching right there should no problem running nix build --override-input CHaP _repo. I can make an aggregate package with all the package components and that would make easy to build the right package.

Exactly. It's not hard to build the repo, override the input, and then issue the appropriate nix command to build everything. The issue is the building and caching, but I think we can solve that with nixbuild.net, potentially.

[michaelpj]

@andreabedini I think we could merge your flake into main even if we can't yet run it in CI? It's still useful for experimentation.

I think I'm okay with having the flake in main pointing to the repo branch and requiring --override-input to get it to point to a newly built version, so long as we document it. I think that's less confusing than having the flake itself in a different branch.

Also I've requested a nixbuild.net key, so we should be able to try that before long also.

[andreabedini]

I am a bit reluctant to put chap/repo as a flake input in the chap repo :joy: I think it would be confusing.

I don't think there's anything stopping us from experimenting with it in the CI (other than lack to time/resources). We need to figure out which package we want to build and then add something like

$ nix build github:input-output-hk/cardano-haskell-packages/builder#hydraJobs.ghc8107.plutus-core-1-1-1-0.library.x86_64-linux --update-input CHaP --update-input hackage-nix --no-write-lock-file

At the moment we need to specify which component to build, maybe I can collect them all into a single derivation.

[michaelpj]

Well, if you're going to put it in another branch you could put it in the repo branch. Then you don't even need an input, you just use the current git repo.

I just think it's quite undiscoverable if it's not in the main branch. That's where I would like to find it, and it's the branch we'll have checked out when we want to use it

I don't think there's anything stopping us from experimenting with it in the CI

True, although if it's in another branch we'll have to do a slightly awkward dance with checking out multiple branches. I guess that's doable though.

I'd be keen to start, since then we can gradually work on getting the revisions in place.

At the moment we need to specify which component to build, maybe I can collect them all into a single derivation.

I had a function like that in the other PR, something like this:

            package-derivations = args@{ built-repo, compiler-nix-name,  pkgname,  version }:
              let hsPkg = (package-project args).hsPkgs.${pkgname};
              in pkgs.haskell-nix.haskellLib.getAllComponents hsPkg;

[michaelpj]

I think this is now pretty much done, I'll make some issues for specific things.