Multiple global package databases

[TerrorJack]

Currently, we hardcode the global package database path to somewhere in the Cabal datadir of the asterius package, and ahc-boot populates the package database by executing the booting process. This is a major source of coffee break sessions in the past: whenever we need to modify something in the boot libs, we'll need to wipe the ghc-toolkit/asterius installation, rebuild and do another round of ahc-boot.

We should have a default global package database path which falls back to the Cabal datadir, but additionally:

• [ ] Allow the source of boot libs to be picked up from the current working source tree, instead of from the datadir of ghc-toolkit (which requires recompilation of ghc-toolkit and asterius)
• [ ] Allow ahc-boot to use a new global package database path
• [ ] Allow ahc-boot to only compile a part of boot libs, e.g. "up to package X", and later resume the compilation of the rest of boot libs
• [ ] Allow ahc, ahc-link, etc to use custom global package paths

In this way, the experience of hacking boot libs can be much smoother. Still, we're free to schedule coffee breaks whenever we want.

[TerrorJack]

iirc something similar is already done in the iohk fork. @hamishmack do you think we can share some work here?

[TerrorJack]

A lot of recompilation issue here is due to needing to support building from sdist tarballs. It's dubious whether sdist support of packages like ghc-toolkit or asterius is really necessary, since we won't do Hackage releases and everyone checks out a git tree and start from there. If we don't use datadir anywhere then there's no need to forcibly rebuild packages to ensure stuff in the datadir is up-to-date.

[hamishmack]

The approach I have tried is to use nix to build the boot libraries from a snapshot of the asterius. The exact version used is pinned here in default.nix. When you run a nix-shell the asterius_bootdir environment variable is set here in default.nix (this triggers ahc-boot to run the boot libraries are not already built for that version).

Inside the nix-shell you can use cabal new-build all to build the different asterius executables (and the new versions should be on the PATH), but it will continue to use boot libs built with the asterius_bootdir.

At any point it should be possible to commit changes, update the pinned commit in default.nix and restart a new nix-shell to trigger ahc-boot to rerun. You could also run nix-build -A asterius-boot or IIRC set asterius_bootdir to a new location and do cabal new-build all && ahc-boot.

I have also been trying out having an asterius/test/asterius-test.cabal and running cabal new-run --with-ghc=ahc --with-ghc-pkg=ahc-pkg on that instead of ahc-link in the unit tests. I think if we do that we might be able to have an asterius/test/cabal.project file and add boot libraries we want to be rebuilt in the packages: section of that file. That way running cabal new-test on one of the tests in asterius/asterius.cabal would trigger just the boot libraries needed by the test to be rebuilt.