Build requests piling up on workers

[vitalybuka]

https://lab.llvm.org/staging/#/builders/sanitizer-x86_64-linux-bootstrap-msan

[vitalybuka]

I believe previously, maybe a few month ago, it was never more than 1-2 requests.

[vitalybuka]

Another example https://lab.llvm.org/buildbot/#/builders/49

It just started to build 3 days old patch. I would expect it picks the most recent one and discard the rest.

[gkistanova]

This has been discussed in discourse starting from this message - https://discourse.llvm.org/t/staging-buildmaster-no-longer-collapses-requests/80500/3. In short - a few days ago we had unusually large number of commits with changes to cmake files. For such commits we request clean builds to make sure if there is an issue it will not be hidden by incremental builders. Build requests with different sets of properties do not collapse.

If this would become a regular problem, I'll rethink of the combining logic.

I quickly checked and this is the case for the builder mentioned here as well.

[vitalybuka]

Is there an option to disable this for a particular builder? For out bots, with sanitizers, many of them 4+ Hour

https://lab.llvm.org/buildbot/#/waterfall?tags=sanitizer

Also they don't run incremental builds, but instead, as suggested, use ccache. So scheduling clean build unnecessary,

[gkistanova]

None of them is marked as doing clean builds.

Could you prepare a patch which either explicitly sets clean=True for these builders, or changes the default in getSanitizerBuildFactory to True, please?

And I'll think of changing the collapse logic to use that information.

[DavidSpickett]

And I'll think of changing the collapse logic to use that information.

We (Linaro) are working on changes to do this.

[DavidSpickett]

We've been able to implement this new rule and it appears to be working. We're going to test it more over the weekend and have changes for review next week.