Open DhanalakshmiDurairaj opened 10 months ago
Hi @DhanalakshmiDurairaj .
This is expected, and a known deficiency in transitions.
Out of curiosity, what's the fundamental problem this causes for you? Obviously rebuilding for no reason is no good and we want to improve on that. I'm curious how concretely you notice this.
Hi @gregestren ,
Thanks for your quick response!
Here, I am sharing the detailed information about the issue that we face:
We are working in Automotive software embedded project. It uses toolchains like GHS and QNX. our objective is to create an output binary image from the source code using these toolchains.
mkimage(
name = "image_file",
binary_a = ":binary_using_GHS",
binary_b = ":binary_using_QNX",
)
Here, I will use the generated binaries from GHS and QNX to build image file. I need to build binary_a and binary_b with one invocation each with different target architectures. I am using bazel transitions. It works as expected. When try to build the same targets individually with same configurations as specified in the command_line_option of bazel transitions, it starts rebuilding.
.bzl file is as follows:
def _impl(settings, attr): _ignore = (settings, attr)
return {
"Platform A": {
"//command_line_option:platforms": "@ghs_compiler//:ccarm",
},
"Platform B": {
"//command_line_option:platforms": "@qnx_compiler//aarch64le",
},
}
multi_arch_transition = transition( implementation = _impl, inputs = [], outputs = [ "//command_line_option:platforms", ], )
def _rule_impl(ctx): binary_a_platform_a = ctx.split_attr.binary_a["Platform A"] binary_b_platform_b = ctx.split_attr.binary_b["Platform B"] img_file=ctx.actions.declare_file("image") ctx.actions.run( executable = ctx.executable.image_tool, arguments=[binary_a_platform_a.file,binary_a_platform_a.file,"-o",img_file.path], ... )
return [DefaultInfo(
files = depset(direct = img_file),
)]
mkimage = rule( implementation = _rule_impl, attrs = { "_allowlist_function_transition": attr.label( default = "@bazel_tools//tools/allowlists/function_transition_allowlist", ), "binary_a": attr.label(cfg = multi_arch_transition), "binary_b": attr.label(cfg = multi_arch_transition), }, )
BUILD file cc_binary( name = "binary_GHS", srcs = ["src/main.c"], deps = [ "//libraries/myOtherLib:myotherlib", ], )
cc_binary( name = "binary_QNX", srcs = ["src/main.c"], deps = [ "//libraries/myOtherLib:myotherlib", ], )
mkimage( name = "image_file", binary_a = ":binary_GHS", binary_b = ":binary_QNX", ) Problems: While executing the targets in modular basis like
To avoid re-building, currently we use bazel run to install the created binaries in the host machine and using the binaries to build image file. By using this approach, we lost dependencies between image build and rest of the builds like GHS and QNX. Any way to achieve our objective by maintaining the dependencies throughout all builds? Thanks & Regards, Dhanalakshmi Durairaj
I see.
Naive followup: why do you need to build the binaries independently (in steps 1 and 2), if step 3 builds them the correct way anyway?
Regarding the inefficiency, you're just transitioning on a platform. I suspect @sdtwigg 's "platforms in output paths" work could help with this: https://github.com/bazelbuild/bazel/commit/293646c85a718686ce1d7a7ff6d1dd76256cb9bb. It should eliminate the differences caused by transitions that only set --platforms
.
Read through the commit and let me or @sdtwigg know if you're able to experiment with this.. These are new changes so you'd need a fresh version of Bazel (I'm not sure if it made it into Bazel 7.0).
Hi @gregestren Naive followup: why do you need to build the binaries independently (in steps 1 and 2), if step 3 builds them the correct way anyway? In our development team we are having more than one developers working on the same project which means that developer 1 would need a final image (in our case step 3 build) and at the same time developer 2 would need a intermittent build either step 1 (GHS) or step 2 (QNX). In the use case mentioned our expectations is to reuse the generated artifacts instead of again re build.
hi @sdtwigg and @gregestren Currently, I am bazel 6.4.0. I tried the flag (--experimental_override_name_platform_in_output_dir) along with --experimental_platforms_in_output_dir it throws error as unrecognized option, can you please help to explore on this to resolve on this issue.
I need to build multiple targets with one invocation each with different target architectures. I am using bazel transitions. It works as expected. When try to build the same targets individually with same configurations as specified in the command_line_option of bazel transitions, it starts rebuilding. .bzl file is as follows: `def _impl(settings, attr): _ignore = (settings, attr)
multi_arch_transition = transition( implementation = _impl, inputs = [], outputs = [ "//command_line_option:platforms", ], )
def _rule_impl(ctx): binary_a_platform_a = ctx.split_attr.binary_a["Platform A"] binary_b_platform_b = ctx.split_attr.binary_b["Platform B"] files = binary_a_platform_a.files.to_list() + binary_b_platform_b.files.to_list()
my_custom_multi_arch_rule = rule( implementation = _rule_impl, attrs = { "_allowlist_function_transition": attr.label( default = "@bazel_tools//tools/allowlists/function_transition_allowlist", ), "binary_a": attr.label(cfg = multi_arch_transition), "binary_b": attr.label(cfg = multi_arch_transition), }, )
BUILD file is as follows:
cc_binary( name = "binary_platform_a", srcs = ["src/main.c"], deps = [ "//libraries/myOtherLib:myotherlib", ], )cc_binary( name = "binary_platform_b", srcs = ["src/main.c"], deps = [ "//libraries/myOtherLib:myotherlib", ], )
my_custom_multi_arch_rule( name = "multiarch_build", binary_a = ":binary_platform_a", binary_b = ":binary_platform_b", )`
To build both binary targets (each configured with different architectures):
bazel build -s multiarch_build
It builds both binary targets as expected.
When I try to build an individual target again with the below command:
bazel build -s binary_platform_b --platforms=@external_repo//platforms:ccarm
It starts rebuilding. It does not use cached artefacts. It shows the below INFO
INFO: Build option --platforms has changed, discarding analysis cache.
I am expecting to use the artefacts from the actions of the target (multiarch_build)
How to avoid rebuilding? Here, platforms specified in transitions and in the command line to build individual targets are same. How it rebuilds and how to avoid? Where to change the code or other configurations to avoid rebuilding?
Pls help me to fix this issue.