Migrate Mimir to distroless Docker image

[pracucci]

We should migrate Mimir to a distroless Docker image, to reduce the attack surface.

The main pushback towards this move was because it's then very difficult to debug a live container (e.g. look at files on disk or sockets), but in recent K8S versions we could use kubectl debug (doc) to add a debugging container to a running pod.

[replay]

Nice, I didn't know about kubectl debug before. The debuggability would have been my only reason to not use a distroless image, so this reason is void now.

[pstibrany]

I like the idea, but kubectl debug and ephemeral containers are new feature in Kubernetes (marked as stable since 1.25, released in late august 2022), and not everyone has it available yet. For example it is not enabled in our dev GCP cluster. How long should we wait before proceeding with this?

[pracucci]

How long should we wait before proceeding with this?

This issue originates from a conversation I had with Thomas (head security at Grafana) and Bryan.

My idea is to publish both alpine and distroless images for Mimir (and GEM) so that users / customers can choose. Moreover, we should also pubilsh a mimir-debug image to use for debugging purposes (with all our tooling inside, e.g. s3/gcs/azure clients). At Grafana we could slowly migrate to distroless, and share our learnings (documentation, improve the debug image, ...).

WDYT?

[pstibrany]

My idea is to publish both alpine and distroless images for Mimir (and GEM) so that users / customers can choose. Moreover, we should also pubilsh a mimir-debug image to use for debugging purposes (with all our tooling inside, e.g. s3/gcs/azure clients). At Grafana we could slowly migrate to distroless, and share our learnings (documentation, improve the debug image, ...).

WDYT?

Publishing both sounds like a good idea. Internally we don't use Mimir docker images in any way, so that's a separate conversation.

[tina-junold]

I saw that this ticket is still open. With https://github.com/grafana/mimir/pull/8204 the switch to distroless was made. Also the grafana/mirmir-alpine is kept, but i don't see any grafana/mimir-debug image. Since Mimir also uses memberlist, in debug environment a shell is required. See my issue here https://github.com/grafana/pyroscope/issues/3478. I hope you find a company wide solution for this. Maybe something like grafana/mimir:latest-debug :-)

[dimitarvdimitrov]

There are some docs in the runbooks for how to run a debug container to a Kubernetes Pod. Does this solve your problem? Or are you specifically looking for a Mimir container which has utilities bundled?

[tina-junold]

@dimitarvdimitrov While i'm using mimir in k8s, this request is not related to k8s. I've created a repository for collecting telemetry data locally. As you can see, that i added custom entrypoints to mimir and pyroscope, since they using memberlist protocol an requiring an IP. I've used the entrypoint to get the docker host ip for this, but since no sh/bash is available, i need to set the ip hard, which can be failing if another system does not use the ip (e.g Macs having another system).

Therefore, the question is, if it is possible to publish a "debug" container witch shell support.

[dimitarvdimitrov]

interesting. Building an image shouldn't be too difficult. Mimir is statically compiled, so just dropping the binary from the existing image or the GH release artifacts into a new image should work. You can verify with the current Dockerfile for the distroless image.

The reason why we went the distroless path is because it's less of a maintenance burden to patch CVEs on unrelated packages. I think nothing related to that has changed, so it's unlikely that we reintroduce an image with a heavier base image.

[tina-junold]

I've updated the repo using two custom Dockerfiles using the binaries of the original containers and adding the entrypoints directly in these containers. Works so far...

[Footur]

I've updated the repo using two custom Dockerfiles using the binaries of the original containers and adding the entrypoints directly in these containers. Works so far...

@tina-junold How is your experience since then? If all goes well, would you like to create a pull request with your changes?