Evaluate attributes in parallel (not just build)

[michaelpj]

I'm not totally sure if this is what's happening, but I think hercules will only try and evaluate one attribute at a time.

However, at least for me it's fairly common that the various attributes have quite different build requirements, e.g. I have top-level darwin and linux attributes which (naturally) use completely disjoint builders. So there is work for all my builders, if they knew about it!

I don't have a clear idea of how you would implement this (does the Nix evaluator even let us do things in parallel?).

[roberth]

This is about parallelizing evaluation if I understand correctly? Normally this shouldn't be necessary, because most time is spent on building rather than evaluation and evaluation does not block the cluster in the way that it blocks with some other CIs. There's some room for optimization, including parallelism. Could you point out a project that has suboptimal evaluation performance?

[michaelpj]

Sure: you can see it on plutus for example.

I have essentially this:

{
    darwin: <something that requires IFD>;
    linux: <something that requires IFD>;
}

What I think is happening is:

1. Start evaluating darwin.
2. Build dependencies of IFD in parallel.
3. Finish evaluating darwin, start evaluating resulting derivations in parallel
4. Start evaluating linux
5. Build dependencies for IFD in parallel.
6. Finish evaluating linux, start evaluating resulting derivations in parallel

In particular, step 4. doesn't start until step 2. is done, although it can happen at the same time as step 3.

So while we're doing step 2. the linux builders are idle, even though there's linux work to be done later (step 5).

This can be an issue when you need to build lots of stuff for IFD, but I've also noticed it when the IFD just seems to take a while: I see the job page gradually adding darwin jobs, and it's not until they're all added that it starts adding the linux ones.

[dhess]

This was also our experience while using Hercules CI. Nearly all of our projects use IFD and evals on Hercules were taking quite a bit longer than on our Hydra, when we tried racing them on the same job.

[roberth]

I am aware of a performance issue. I will look into it after my current sprint to improve flake and "polyrepo" support.

[brainrake]

bump. big issue in mlabs-haskell, as we use a lot of IFD.

[roberth]

Implemented in https://github.com/hercules-ci/hercules-ci-agent/pull/428, release 0.9.4, May 2022.

[roberth]

(does the Nix evaluator even let us do things in parallel?).

It's not designed to work in parallel. #428 stops when IFD is encountered, continues with other attributes, and revisits when progress was made. (ie concurrency, not parallelism)