vitarb
commented
5 years ago To give a little more context, this feature would be extremely useful and will help to produce a list of changed targets between two commits on CI allowing correctly scoped builds (instead of //...) and also allowing other pre-land optimizations. As previous discussion suggests there is no easy way to detect changes in the build graph when it comes to modifications in starlark files (WORKSPACE, BUILD, *.bzl, etc) Although it's possible to implement this logic outside of bazel, it would be prone to either false positives or false negatives and can also be rather slow.
For example I've built reasonably working prototype that:
- Runs
bazel query deps(//...), parses it's output and builds in-memory representation of the dependency graph. - Finds a list of changed files (using git) and for each: a) for source files - finds all reverse dependencies in the build graph (easy case). b) for starlark files - parses the file using bazel's starlark parser at both revisions, hashes all expressions and finds which ones changed.
- Translates each changed call expression or assignment into bazel path that it can be referenced from (e.g. //src/foo:foo for BUILD files) and (@foo for WORKSPACE file) and find all reverse dependencies of that marking them as changed.
- Native rules are just assignments in the starlark file and are not represented as rule-inputs in the build graph so they require special logic to handle.
Although it works fast and requires only one bazel query it may lead to possible false negatives:
- It might be not able to handle some edge cases when there is no connection between starlark definitions and dependency graph (for example if you change go version defined in the go toolchain it won't detect any changes as toolchain info is not present in the dependency graph, work-around for that would be extracting go version into a variable and using it as a tag in the io_bazel_rules_go which comes as a dependency for all go rules)
- Identifying if given starlark rule has changed requires handling various scenarios in the starlark syntax such as macros, assignments and call statements. It's relatively easy to miss some edge case.
We've also considered another approach, which relies on query of rbuildfiles for each changed starlark file but what makes it not acceptable is that any change to the WORKSPACE file would trigger pretty much full repo rebuild as all third-party dependencies are defined there. In addition to false positives it also has scalability issues as it needs to issues a separate query for each changed starlark file which runs ~0.5 sec, meaning that reasonably sized refactoring affecting 1000s of build files can be computing changed targets for 10s of minutes.
Are there any reasons preventing us from adding this feature into bazel core and printing checksum next to each rule in the dependency graph?
Globegitter
lberki
purkhusid
ericfelly
haxorz
robbertvanginkel
meisterT
mmlac
tokudu
linzhp
tinder-maxwellelliott
rohansingh
janakdr
codesuki
github-actions[bot]
Description of the problem / feature request:
Buck has the concept of rule keys, which we can obtain by running
buck targets --show-rulekey //.... We'd love to see similar feature in Bazel.Feature requests: what underlying problem are you trying to solve with this feature?
We need to see what targets are changed, thus need to rebuilt and test, from one revision to another. In order to keep master green at scale, we need to build and test diffs in parallel. Getting the rule key for each target help us decide what diffs are independent and safe to build and test in parallel.