Example for using different environment files for different platforms

[jiawen]

I'm going down a Bazel/conda monorepo rabbithole. I did a bunch of reading and it seems the existing rules, by passing kwargs should be able to support the use case where we have different platforms requiring different environment files (e.g., GPU support on Linux but not on Mac).

I think this is possible with a combination of:

• Bazel platforms
• Passing --platforms=... as Configurable attributes.
• and Toolchains passing platform constraints to register_toolchains.

Shall I set up a simple example with a test?

[spietras]

Can you elaborate more on how would you see the flow of working with that kind of setup with different environment files? How the user would use the rules in WORKSPACE to set things up?

[jiawen]

At a high level, the use case is clear: for a multi-platform build, not all versions of dependencies are supported on all platforms (say, Linux vs MacOS).

In the WORKSPACE file, we would have one call to conda_create for each environment file:

conda_create(
  name = "linux_env",
  environment = "@//:linux_env.yml"
)

conda_create(
  name = "macos_env",
  environment = "@//:macos_env.yml"
)

Then register the toolchains with platform constraints:

register_toolchain(
  py3_env = "linux_env"
  target_compatible_with = "@platforms//os:linux"
)

register_toolchain(
  py3_env = "macos_env"
  target_compatible_with = "@platforms//os:macos"
)

Python toolchain resolution will take care of the rest.

I dug a little further - to support this, we need to plumb target_compatible_with through toolchain.bzl:toolchain_rule into the call to toolchain.

[spietras]

I like it. Just let's keep in mind to have the target_compatible_with optional so by default it's compatible with anything.