search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.1k stars 4.04k forks source link

Parallelize repository up-to-dateness check #20869

Open lberki opened 9 months ago

lberki commented 9 months ago

Description of the feature request:

External repositories need to verify that their file dependencies are still up-to-date. This happens on one thread in RepositoryFunction.verifyMarkerDataForFiles(). This means that if a repository has many files to check, we are leaving a lot of performance on the table. It would be nice if we used all the parallelism available (it's mostly checksumming, so it's not obvious whether it's I/O or CPU-bound, but more parallelism helps either way, the question is just how much more)

The obvious way to do this is to have a little thread pool, but we already have many little thread pools (e.g. TreeArtifactValue.VISITOR_POOL which does something similar), so maybe some consolidation is in order.

Which category does this issue belong to?

No response

What underlying problem are you trying to solve with this feature?

Make null builds faster.

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

`7dd569f1b0166cbf651789d91ce192d8bde44267`

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

lberki commented 9 months ago

Assigning to @meteorcloudy for routing and maybe as a personal mental health coding task.

meteorcloudy commented 9 months ago

/cc @Wyverald