write guide section on: productionising, containers and dev workflow

[clux]

There are many many many different ways of packaging rust applications into docker to make a nice workflow. Many of them have tons of annoying cache-caveats (some that only work on CI, some that only work locally).

I would like to have a good guide with some pros/cons discussions on deployment choices and that optimise for a few different metrics (parenthesised):

• [ ] standardised rust image (readability)
• [ ] standardised rust image as managed by CI providers like circleci (CI image locality)
• [ ] above but with cargo-chef (CI cache time)
• [ ] musl cross compilation (docker image size, less direct C dependencies)
• [ ] arm cross compilation (cloud compute costs via graviton)

Most of these have some annoying side effects:

• rust image -> slow iteration time locally and on CI
• CI provided rust image -> faster on CI, but potential differences between local build and CI
• cargo-chef -> large, inscrutable docker images, and still awkward for local builds
• musl cross -> potential musl compat issues and static toolchain upgrade problems (C deps in image)
• arm cross -> ??? need to know more ??? looks very new

Additionally it would be nice to also discuss some stuff that can (probably) be used with most of the above choices:

• [ ] docker buildkit caching (e.g. version-rs's Dockerfile) (hard to get right on CI, but perhaps possible now)
• [ ] distroless for security benefits
• [ ] how to pave over awkward build invocations with tilt / skaffold
• [ ] something about benefits of compromising on mono-stack and using different images for linting

With this we can probably close kube-rs/kube-rs#332

[kazk]

large, inscrutable docker images, and still awkward for local builds

Can you elaborate? I've been using cargo-chef with your clux/muslrust:stable image in a multi-stage build. I don't think I noticed anything different for local builds.

docker buildkit caching (e.g. version-rs's Dockerfile) (https://github.com/kube-rs/version-rs/pull/4 to https://github.com/kube-rs/version-rs/pull/5 right on CI, but perhaps possible now)

I'm pretty sure GitHub Actions cache is working now with cache-from: type=gha and cache-to: type=gha,mode=max. See logs in https://github.com/kazk/cnat-rs/runs/5263785000.

[clux]

Can you elaborate? I've been using cargo-chef with your clux/muslrust:stable image in a multi-stage build. I don't think I noticed anything different for local builds.

Oh, hm. I remember struggling to get it to do local caching, but not sure what the issue was. I perhaps did not give it enough time though. Remember being put off by the amount of FROM statements it needed.

They do also have a fat disclaimer on their readme saying "warning cargo-chef is not meant to be run locally" (EDIT: although i guess they mean outside a container, locally)

pretty sure GitHub Actions cache is working now

ah, that is great news! i remember battling with it for a weekend and ended up frustrated. It feels like that would be a great thing to recommend for caching since it doesn't add multiple extra FROM statements and works both locally and on CI.

[kazk]

It feels like that would be a great thing to recommend for caching since it doesn't add multiple extra FROM statements and works both locally and on CI.

To be clear, cache-to/from: type=gha caches the layers automatically using GitHub Actions cache service, and it'll make the layer cache work like when you're building locally. It won't help if the image is not taking advantage of layers (e.g., by using cargo-chef). Multiple FROM/RUN in Dockerfile is necessary to make efficient layers.

[clux]

True. I guess what I mean is that: it would be ideal if you can take advantage of the gha caching in conjunction with buildkit's mountable cache to avoid having to forward all this state between these layers to make it as efficient.

I think that's probably what didn't work last I tried (caching the mounted caches).