Partition pre-build of devcontainers into LLVM/MLIR vs. heir-specifc

[AlexanderViand-Intel]

Currently, the HEIR codespaces are defined via monolithic devcontainer.jsons that do everything from basic dev-environemnt setup to building LLVM/MLIR and HEIR in what is effectively a single dockerfile.

This works, but is somewhat wasteful as we should only need to re-build LLVM/MLIR whenever there's been an LLVM_INTEGRATE commit. This not only means burning unnecessary action runner minutes, but also introduces a huge latency between updates to main and the codespaces being up-to-date.

We can solve this by splitting the build into two images, one for LLVM/MLIR and one for the HEIR specific stuff (and niceties such as vscode settings/etc).

Afaik, this requires complicating the system a bit, but not too much so: we'll need to create our own action that builds and publishes a docker/devcontainer image whenever the current LLVM_COMMIT isn't the one of the last published image, and then the devcontainer should just be a relatively lighweight layer on top of that image. I think the key thing to research is where and how to publish that image.

@makslevental : as discussed at the MLIR workshop, it should be possible to recycle this prebuild LLVM/MLIR image for other projects, too, as long as they're fine with following our llvm integrate.

EDIT: I'm thinking of doing this with the cmake devcontainer first, because that already has a much clearer separation between LLVM/MLIR and HEIR builds. I'd love some input from the bazel experts to see if there's a nice way to do this for bazel, too (@j2kun @asraa?)

[asraa]

it should be possible to recycle this prebuild LLVM/MLIR image for other projects, too, as long as they're fine with following our llvm integrate.

The tensorflow project follows our exact setup, so they also have copybara publish updates to the LLVM commit whenever it's merged in Google.

I'd love some input from the bazel experts to see if there's a nice way to do this for bazel, too (@j2kun @asraa?)

Bazel caching will end up managing itself, but we can re-build the LLVM repo on the new commit triggers (bazel build @llvm-project//...) but in order for the heir build to reuse that, it will have to make sure anything else in the environment hasn't changed.

[makslevental]

Bazel caching will end up managing itself

can bazel cache be stored/serialized and then used to pre-populate an empty cache? E.g., in a docker container (or a fresh GHA instance)?

[j2kun]

Bazel caching will end up managing itself

can bazel cache be stored/serialized and then used to pre-populate an empty cache? E.g., in a docker container (or a fresh GHA instance)?

This is what we do already for our builds. If there's no LLVM commit, incremental CI actions are relatively fast.