Is your feature request related to a problem? Please describe.
The POSIX toolchain provided by rules_sh tracks and exposes the tools it bundles as strings, paths to binaries installed outside of Bazel. The common use-cases are either the Nix use-case, where these paths will point to a Nix store path, or the default use-case, where the toolchain setup looks the tools up on PATH and then stores the resolved absolute paths.
So long as we use specific tools by absolute paths this is not an issue, e.g. so long as we explicitly invoke $(POSIX_GREP). However, in some use-cases we need to provide the tools bundled by the toolchain in a PATH variable to another binary or script. In this case the PATH may cover more tools than are really meant to be captured by the toolchain. For example, if the default POSIX toolchain found the tools in /usr/local/bin, then PATH will also expose other tools installed under /usr/local/bin, e.g. /usr/local/bin/clang or /usr/local/bin/my_custom_binary_that_should_not_be_known_to_bazel. This breaks hermeticity.
Describe the solution you'd like
This can be solved by symlinking the discovered tools into the Bazel sandbox and only exposing them to build actions through these symlinks. I.e. build actions would no longer invoke /usr/local/bin/grep, but instead something like external/posix/grep, note the relative vs. absolute path. These symlinks would also need to be declared as explicit inputs into the build action.
Note, on Windows symlinks may not be available. In that case we can instead use shims. For example by using Scoop's Shim.
Notably this enables further use-cases that are currently not supported. For example, with this approach one could fetch a busybox release, or some other bundle of shell binaries, and generate a POSIX toolchain based on it.
This approach is already implemented here and would only need to be upstreamed into this repository. A Windows variant is implemented here, that one is specific to that particular use-case, but the concept can be generalized.
Additional context
The approach implemented here is more general. It is not restricted to tools for a POSIX toolchain, but applies to any bundle of binaries. Similarly, the same module contains an implementation for a cc_library bundle.
[x] Introduce a utility to create Shims on Windows (to provide an alternative to symbolic links, which are not generally available on Windows).
Addressed in https://github.com/tweag/rules_sh/pull/26 by adding the create_shim utility.
[ ] Introduce a utility to collect tools from PATH (e.g. using repository_ctx.which) and import them into an external Bazel workspace (e.g. using repository_ctx.symlink or create_shim as appropriate). (Beware of false friends)
[ ] Define a backwards compatibility strategy for a transition of the current POSIX toolchain to a variant based on File objects.
[ ] Implement a POSIX toolchain based on sh_binaries.
[ ] Implement a repository rule to import POSIX tools from PATH using symlinks or Shims as appropriate.
Is your feature request related to a problem? Please describe.
The POSIX toolchain provided by rules_sh tracks and exposes the tools it bundles as strings, paths to binaries installed outside of Bazel. The common use-cases are either the Nix use-case, where these paths will point to a Nix store path, or the default use-case, where the toolchain setup looks the tools up on
PATHand then stores the resolved absolute paths.So long as we use specific tools by absolute paths this is not an issue, e.g. so long as we explicitly invoke
$(POSIX_GREP). However, in some use-cases we need to provide the tools bundled by the toolchain in aPATHvariable to another binary or script. In this case thePATHmay cover more tools than are really meant to be captured by the toolchain. For example, if the default POSIX toolchain found the tools in/usr/local/bin, thenPATHwill also expose other tools installed under/usr/local/bin, e.g./usr/local/bin/clangor/usr/local/bin/my_custom_binary_that_should_not_be_known_to_bazel. This breaks hermeticity.Describe the solution you'd like
This can be solved by symlinking the discovered tools into the Bazel sandbox and only exposing them to build actions through these symlinks. I.e. build actions would no longer invoke
/usr/local/bin/grep, but instead something likeexternal/posix/grep, note the relative vs. absolute path. These symlinks would also need to be declared as explicit inputs into the build action.Note, on Windows symlinks may not be available. In that case we can instead use shims. For example by using Scoop's Shim.
Notably this enables further use-cases that are currently not supported. For example, with this approach one could fetch a busybox release, or some other bundle of shell binaries, and generate a POSIX toolchain based on it.
This approach is already implemented here and would only need to be upstreamed into this repository. A Windows variant is implemented here, that one is specific to that particular use-case, but the concept can be generalized.
Additional context The approach implemented here is more general. It is not restricted to tools for a POSIX toolchain, but applies to any bundle of binaries. Similarly, the same module contains an implementation for a cc_library bundle.
Tasks
sh_binariesrule.create_shimutility.PATH(e.g. usingrepository_ctx.which) and import them into an external Bazel workspace (e.g. usingrepository_ctx.symlinkorcreate_shimas appropriate). (Beware of false friends)Fileobjects.sh_binaries.PATHusing symlinks or Shims as appropriate.sh_binaries. https://github.com/tweag/rules_sh/issues/46