Basic project setup no longer fully cached

[amesgen]

When creating a basic default project (e.g. via github:NixOS/templates#haskell-nix, but removing all cross-compilation stuff) and updating all inputs via nix flake update, entering a shell is no longer fully cached. Rather, I have to compile cabal-install and nix-tools myself beforehand, which a significant amount of time.

I think that

• ensuring that all IFD dependencies and standard tools like cabal, haskell-language-server you get via tools are cached, and
• explicitly documenting what you can (not) expect to be cached apart from the GHCs themselves

would be very nice for UX.

I speculate that this change in behavior might be due to the "Update Hackage and Stackage" being created via GH Actions, which in turn means that GH Actions will not be run on them (which leaves no chance for the cache being populated).

[billksun]

Not only that, I tried changing to use GHC 9.2.5 and it's re-building all the GHCs. I am using unstable nixpkgs.

[EncodePanda]

I can get GHC binary cache hits for 9.2.5 but I have to rebuild all the dependencies.

[billksun]

I can get GHC binary cache hits for 9.2.5 but I have to rebuild all the dependencies.

Ah, so that's what's happening. I was scratching my head as nix path-info is saying GHC is cached, but it still takes 1.5 hours before I hit the stylish-haskell error.

[amesgen]

FTR: The zw3rk cache seems to contain everything necessary:

https://github.com/input-output-hk/haskell.nix/blob/ee6e1d72e3bb17c4761fd9eb37aecca2174e4cc9/.github/workflows/updates.yml#L17-L18

Is this just an intermediary measure until cache.iog.io can be populated again, or is this a more permanent thing? If so, it might make sense to mention this in the "Getting started" pages, as it would greatly improve the experience of using recent haskell.nix commits. Or maybe @angerman would not like these caches to be used by the general public?

[peterbecich]

Attempting to build a project with GHC 9.4.4

error: a 'aarch64-darwin' with features {} is required to build 
'/nix/store/gi38vzmbxhjrhl19jz4afvrn5lqw2srn-hadrian-source.drv', 
but I am a 'x86_64-linux' 

This error does not occur for the same project with GHC 9.2.5.

Could this be explained by hadrian-source for aarch64-darwin and GHC 9.2.5 being cached, but not for GHC 9.4.4?

[klarkc]

I was wondering why so huge build times, thanks @amesgen for mentioning the workaround. This cache is public? Can/should I use it in nix-templates?

[klarkc]

I believe it's public, because it's being recommended in plutus-apps

[cumber]

So is it just intended that Haskell.nix users should be using cache.zw3rk.com now? If so, shouldn't that be in the documentation, templates, etc?

Or is hydra.iohk.io intended to contain everything needed to start going? Currently without cache.zw3rk.com a basic ghc 9.2 or 9.4 project starts by building ghc 8.8.4 from source (presumably in order to then build ghc 9.x from source); surely even if the recent compilers aren't in the cache the older one used to build it would be?

[peterbecich]

@cumber , I don't have an answer to your question but an updated Flake might be cached https://github.com/NixOS/templates/pull/61

I have just tested this and it appeared to pull GHC 9.2.6 from IOG. Would you set the project to 9.2.6 and nix flake update?

[cumber]

@peterbecich I just initialised a project from your update to the template (via nix flake init --template 'github:NixOS/templates?rev=e7a5f0e307f19fb688b063b01b38656cd680753b#haskell-nix').

At that point nix develop got the source for ghc 9.2.6 from cache.iog.io. But at least it started building 9.2.6, not building 8.8.4 (so that it could use it to build 9.2.6?), so presumably the 8.8.4 build either was fetched from the cache or matched up with a derivation I had already built locally so I could reuse it. After a nix flake update it did pretty much the same thing (it just spent longer evaluating and downloading hackage snapshots and such before it got to the ghc 9.2.6 build).

[dminuoso]

I can also confirm that at least the final GHC is not cached anymore. I've tried every GHC version specified in the ci/ci.nix for x86_64-linux and I'm not getting cache hits.

Either this is a bug, or the cache is not served anymore. Whatever it is, we should document it in the guide, otherwise users will have a reasonable expectancy of not compiling GHC themselves.

[angerman]

@dminuoso which cache are you using? cache.zw3rk.com should have pretty much all of it. A bunch of cross compilers are missing though in master.

It is building /a lot/ of compilers right now. https://ci.zw3rk.com/jobset/input-output-hk-haskell-nix/master

Everything that's built on there ends up in the cache.zw3rk.com.

[dminuoso]

@angerman Hi Moritz,

If I follow the Getting Started guide I am told to use https://cache.iog.io. And that cache seems to not be populated by this CI anymore.

Should new users that hope to make use of cache hits use https://cache.zw3rk.com now? If yes, that should be reflected in the Getting Started guide. Further some note about who is responsible/populates this cache (this is an alternate but official IOG cache? Is it someones personal cache?) would be nice, as it helps risk assessment when adding this as a substituter.

If not, then at least some acknowledgement that either the CI or cache.iog.io is broken would be nice as well as a note in the guide mentioned above.

[cumber]

It would be really good to get some sort of official statement (i.e. in the docs) that end users should be using cache.zw3rk.com, and whether this is a temporary stop gap or an intended long-term situation.

It just feels weird that we can only find out about this extra cache (whose domain and key name don't obviously have anything to do with IOG or Haskell.nix) from comments from individuals scattered throughout the bug tracker, and have to decide whether we trust that or whether it could possibly be an attempted attack trying to get people to trust binary artifacts from a dodgy cache.

[angerman]

We will try to populate cache.iog.io. zw3rk CI and cache are run by me. There is only so much I'm willing to pay out of pocket for this.

If this ends up being official everywhere, I won't be able to pay for it; hence I'm a bit reluctant to publish this too much.

[domenkozar]

We will try to populate cache.iog.io. zw3rk CI and cache are run by me. There is only so much I'm willing to pay out of pocket for this.

If this ends up being official everywhere, I won't be able to pay for it; hence I'm a bit reluctant to publish this too much.

I'm happy to sponsor the cache at https://cachix.org, let me know what account and the limit.

[masaeedu]

This is a bit of an ancillary issue, but is it possible to set up some kind of CI system so that commits on main are certified (probably through a tag) to have the corresponding closure available on cache.iog.io (or wherever this ends up being located)? This way users can stick to the latest version of haskell.nix for which they can expect to get substitutes.

The benefit here is that if things are broken for some reason it shows up as CI failures and the absence of new tags, rather than as time consuming builds on user's machines. The latter requires careful attention to even identify as a problem, and I can't see an easy way to automate the process of surfacing the lack of substitutes as an error, since a given project derivation will typically have a mix of Haskell infrastructure dependencies that should be cached, and project-specific dependencies that cannot be.

I am very grateful for the work that has gone into haskell.nix and have been happily using it for several years, but the routine unpredictability of entering my shell and finding out whether I'll be spending the next 20 minutes hacking on my project or building sundry derivations and trawling the issue tracker to find out why is pretty taxing.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[RichardWarfield]

Let me preface this by saying a big thanks to the sponsors and developers of haskell.nix, which has been a fantastically useful project for us; what I say here is not meant to cast any shade.

But as the documentation alludes to, configuring working binary caches for GHC is really a must if this library is to be useful. It's literally the first thing the "Getting Started" section tells new users to do. Without cached GHC builds you often end up building not one but several versions of GHC, and each build can be a multiple hour affair even on speedy hardware.

What's worse, there is not (as far as I can tell) any way to know which GHC versions are cached for each revision of haskell.nix without trying and waiting to see whether it starts building GHC. The list of supported versions hasn't been updated since last year and ends at 9.2.2.

If there is anything I can do to help get builds in a cache somewhere I am happy to try, though I'm a moderately proficient Nix user at best.

[emeinhardt]

We will try to populate cache.iog.io. zw3rk CI and cache are run by me. There is only so much I'm willing to pay out of pocket for this. If this ends up being official everywhere, I won't be able to pay for it; hence I'm a bit reluctant to publish this too much.

I'm happy to sponsor the cache at https://cachix.org, let me know what account and the limit.

Did this end up happening?

[domenkozar]

I'm still happy to do that, we've recently shipped a command that imports a cache from S3 to help with the migration.