Proposal: Run flux as unprivileged user

[dkerwin]

The official docker images and Kubernetes manifests run flux as root user. Running as root is not a requirement for flux to operate and I would prefer to run with minimal privileges. Switching to a unprivileged user-account would require these steps:

• Add user account in Dockerfile
• Set USER in Dockerfile
• Update flux to handle ssh files differntly

The main issues that would result from the privilege drop is that flux will no longer be able to update the permissions of ssh-related files. Here's what I propose as a solution:

1. Mount custom ssh-config to a location outside of /home/flux/.ssh
2. During the startup of flux copy the identity and other configs to /home/flux/.ssh
3. Update permissions of /home/flux/.ssh and it's contents

What are your thoughts on this?

[squaremo]

Copying files that are mounted on startup is not adequate; the daemon needs to see changes to the mounted volumes, especially the SSH key. It might be workable just to change the location of these things so that they fall inside /home/flux, and are owned by the flux user. If it's not possible to mount volumes such that they are readable by the flux user (and have the permissions as desired by SSH), this is a non-starter.

The other thing I can think of that may be affected is the access to /var/run/kubernetes, which lets flux connect to the API server. From a quick kubectl exec -- /bin/sh it looks that's world-readable; but worth checking.

[stefansedich]

I have picked this up and am having a go at getting to a working solution that will work for everyone and so far I have a working proof of concept for the flux operator which is working for git sync and also appears to handle gpg signing/verification fine.

https://github.com/weaveworks/flux/compare/master...stefansedich:non-root

The approach consited of:

1. Use an init container, that will copy the secret/configMap values to another memory volume setting the correct ownership and permissions on the keys.
2. Move some things from /root into the /home/flux home folder.

One downside with this approach and as mentioned by @squaremo we lose the ability to simply update the ssh/gpg secrets and have things update and we would require the user to restart the pod for things to take affect. However is this a trade-off we are willing to accept so that we can run as a non-root user?

Hoping to start a discussion now that I have something up and running and before I go sink any more time into it and if we decide pod reboots are 100% unacceptable I can then continue to dig further.

[jacobsin]

Verified just now this approach works - thanks @stefansedich

Although I really have hoped the git-key secrets mount to have the correct 0400 permission at the first place, then there is no need to chmod at the first place , but think I am hitting another open issue with openshift https://github.com/openshift/origin/issues/16424.

Btw, I got this to work without initContainer but just lifecycle.postStart hook

[stefansedich]

Thanks @jacobsin I have been flat out with other things and have not touched this one in a while, I really want to get back to it if there is some interest to get it done.

One issue with my current solution that I wanted to solve was ssh and gpg keys should be updated when a secret changes, obviously my solution breaks that as it does the copy on startup and does nothing when the underlying secret changes down the road.

It would be easy if I could just chmod the secret to 0400 on first read, but with them not being writeable it was the reason why I went the copy on startup route chown to the non-root user and finally chmod 0400 them.

I will see if I get some time this week to play around with a few more ideas I had so this one can be wrapped up.

[ianmiell]

This is a (soft) requirement for us (we can work around it by using resourcequotas to prevent tenants from abusing flux's access to the privileged sccc).

If I can help push this forward I can look at donating some of my free time. LMK.

[goober]

There is an open issue in the Kubernets project that addresses the issue of always setting the root as owner of secret mounts. Could be worth following that thread. https://github.com/kubernetes/kubernetes/issues/81089

[jMarkP]

Hi all, I'm hoping to use flux in our Kubernetes environment, and non-root running is a hard requirement in our setup so I'm keen to help with this effort if I can.

I have a WIP proposal that seems to allow non-root running for me locally and wanted to get some feedback on the approach before I work it up to a full PR.

To get around the requirement for the private key to have 0400 perms so that ssh-keygen can extract the public key from it, my change adds a check for an existing public key (mounted alongside the private key) and uses that if it exists. This allows users who want to run as non-root to pre-generate an ssh key pair through some other mechanism and supply it ready made to flux.

My WIP is here. Clearly this needs more work (tests, documentation etc.) - if you think the approach seems sensible I'd work on productionising it. I can run this locally with the pod running as a non-root user and flux happily uses the key to connect to GitHub and sync workloads.

An example script that would generate the pre-filled key secret is

#!/bin/sh

ssh-keygen -q -N "" -f identity
ssh-keygen -y -f identity >identity.pub

kubectl -n flux create secret generic flux-git-deploy-explicit \
  --from-file=identity \
  --from-file=identity.pub 

rm identity identity.pub

[willholley]

@jMarkP did you get any further with this? I'm not sure whether Flux 2 makes the work redundant - if so it would be good to know so that this ticket can be closed out.

[jMarkP]

Hi @willholley - I managed to get this working on that fork I linked to which we've been using internally to run Flux as a non-root user: https://github.com/fluxcd/flux/compare/master...G-Research:UseSuppliedPublicKey. For the time being we've moved away from Flux v1 onto ArgoCD, and will be looking to try out FluxV2 in the future so happy for this to be closed.

[stefanprodan]

Flux v2 controllers run as non-root users with root FS in readonly mode. Please consider upgrading to v2 as v1 will never get this feature.

[kingdonb]

As was explained before, Flux v1 is in maintenance mode and though inroads were apparently made in #3097, adding this feature as a backport is not straightforward, and it was already implemented in the rewritten version of Flux, GitOps Toolkit.

In the interest of reducing the number of open issues not directly related to supporting Flux v1 in maintenance mode down to something manageable, and hopefully you may have moved on to v2 already, I will go ahead and close out this issue for now.