search

fluxcd / source-controller

The GitOps Toolkit source management component
https://fluxcd.io
Apache License 2.0
232 stars 183 forks source link

Alpine image challenges #1400

Open thejosephstevens opened 4 months ago

thejosephstevens commented 4 months ago

I'm running into some challenges as I attempt to adopt Flux. We manage many different clusters across all 3 clouds (EKS, GKE, AKS), and specifically we operate clusters in mixed-trust environments where the domain of many of our clusters isn't fully trusted (meaning that it would add significant inconvenience to our architecture if each cluster had to have auth to its Github sources - this would force us down a route with a repo per cluster, which makes maintenance burden significantly higher).

The design I'm moving forward on right now is using a central cluster with many sources in one Github repository to manage many different k8s clusters. The challenge I'm running into is with auth - in order to authenticate across cloud to EKS, I need to have the aws cli running locally so that can be used as part of the kubeconfig exec step. The trouble here is, in order to run the AWS CLI on Alpine it looks like I may need to actually build it from source or add in glibc and all the other dependencies, which isn't recommended on alpine (the v2 cli isn't distributed through pip, and if you install the linux version directly it doesn't work on alpine). This amounts to a fairly significant burden to follow this model.

As I look at the rest of the industry, it seems like most tools have aligned on Debian-based systems as the standard for Linux, as the -slim variants of those images have narrowed the size gap against alpine to a negligible amount, CVE coverage in alpine is poor, and there's the issues with musl compatibility.

I'm interested in understanding whether there would be appetite for moving over to debian or ubuntu from alpine, as that would make extensibility of the images significantly easier (the story on alpine support is pretty much the same on all clouds, no one is building apk packages for any of their distributables, and anything that depends on C in any way you're in for a bad time).

If there's interest but little maintainer capacity I may be able to get time to take a stab at this, as at this point I'm starting to wonder if that would actually be easier than getting aws cli into alpine, but I'd like to understand if there are other paths I'm not thinking of from others who have solved similar problems, and whether there is interest in this as a direction for flux.

Thanks!

souleb commented 3 weeks ago

For the specific move to a debian base image, I think the best would be to join one of the dev meeting.

Your specific issue though ressemble what is described in https://github.com/fluxcd/flux2-hub-spoke-example.