Linkerd without CNI - run as non-root

[part-time-githubber]

Feature Request

What problem are you trying to solve?

We are not able to run Linkerd with CNI as another CNI we have in the chain does not play well with Linkerds. Hence we want to switch to Linkerd without CNI. But that needs our kube PSPs to allow

• capability NET_RAW
• capability NET_ADMIN
• run as root

We are OK with the first two capabilities confined to the linkerd proxy binary without needing to run as root. That would be acceptable to InformationSecurity team at the place we are trying to run Linkerd. Is it possible to do that?

How should the problem be solved?

linkerd proxy container section added by mutation webhook will need to include something like securityContext: capabilities: add:

• NET_RAW
• NET_ADMIN

the proxy image will need to be built with something like RUN apt-get install libcap2-bin=1:2.25-2 -y --no-install-recommends \ && setcap 'cap_net_raw=+ep' \ && setcap 'cap_net_admin=+ep'

What do you want to happen? Add any considered drawbacks.

Any alternatives you've considered?

Is there another way to solve this problem that isn't as good a solution?

Run all application workloads as Root which opens a big security hole

How would users interact with this feature?

If you can, explain how users will be able to use this. Maybe some sample CLI output?

Kubernetes PSP for application workloads which needs to allow above two capabilities and can still require binary to NOT run as root

[adleong]

Hey @pankajmt! If we could run the proxy-init container as non-root but still with the appropriate net capabilities, we'd be open to it. As far as I know, no one has tried this so I'm not totally sure if it is possible or not. If you are interested in working on this, I'd love to take a look.

[part-time-githubber]

ah yes, it would be the proxy-init container and not the proxy container to be precise. I might start looking into this today.

[part-time-githubber]

Not much luck with attempt 1

[23:26] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt]$ [._.] cd linkerd2
[23:26] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2]$ [._.] git diff
diff --git a/charts/partials/templates/_proxy-init.tpl b/charts/partials/templates/_proxy-init.tpl
index c818e713..e821f9dd 100644
--- a/charts/partials/templates/_proxy-init.tpl
+++ b/charts/partials/templates/_proxy-init.tpl
@@ -44,8 +44,8 @@ securityContext:
   privileged: false
   {{- end }}
   readOnlyRootFilesystem: true
-  runAsNonRoot: false
-  runAsUser: 0
+  runAsNonRoot: true
+  runAsUser: 70000
 terminationMessagePolicy: FallbackToLogsOnError
 {{- if or (not .Values.global.cniEnabled) .Values.global.proxyInit.saMountPath }}
 volumeMounts:
@@ -58,5 +58,5 @@ volumeMounts:
 - mountPath: {{.Values.global.proxyInit.saMountPath.mountPath}}
   name: {{.Values.global.proxyInit.saMountPath.name}}
   readOnly: {{.Values.global.proxyInit.saMountPath.readOnly}}
-{{- end -}}  
+{{- end -}}
 {{- end -}}

[23:28] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt]$ [._.] cd linkerd2-proxy-init/
[23:28] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2-proxy-init]$ [._.] git diff
diff --git a/Dockerfile b/Dockerfile
index 154d9bc..6b1e2a0 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -18,9 +18,14 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
         iptables \
         procps \
+        libcap2-bin \
     && rm -rf /var/lib/apt/lists/* \
     && update-alternatives --set iptables /usr/sbin/iptables-legacy \
     && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+
+RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
 COPY LICENSE /linkerd/LICENSE
 COPY --from=golang /out/linkerd2-proxy-init /usr/local/bin/proxy-init
+RUN setcap 'cap_net_raw,cap_net_admin=+ep' /usr/local/bin/proxy-init
+USER linkerd
 ENTRYPOINT ["/usr/local/bin/proxy-init"]

See this in startup of linkerd control plane

│ 2021/01/12 05:59:33 :; iptables -t nat -N PROXY_INIT_REDIRECT -m comment --comment proxy-init/redirect-common-chain/16104311 │
│ 73                                                                                                                           │
│ 2021/01/12 05:59:33 iptables v1.8.2 (legacy): can't initialize iptables table `nat': Permission denied (you must be root)    │
│ Perhaps iptables or your kernel needs to be upgraded.                                                                        │
│                                                                                                                              │
│ 2021/01/12 05:59:33 Aborting firewall configuration                                                                          │
│ Error: exit status 3                                                                                                         │
│ Usage:                                                                                                                       │
│   proxy-init [flags]                                                                                                         │
│                                                                                                                              │
│ Flags:                                                                                                                       │
│   -h, --help                               help for proxy-init                                                               │
│       --inbound-ports-to-ignore strings    Inbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy. │
│  This has higher precedence than any other parameters.                                                                       │
│   -p, --incoming-proxy-port int            Port to redirect incoming traffic (default -1)                                    │
│       --netns string                       Optional network namespace in which to run the iptables commands                  │
│       --outbound-ports-to-ignore strings   Outbound ports and/or port ranges (inclusive) to ignore and not redirect to proxy │
│ . This has higher precedence than any other parameters.                                                                      │

[part-time-githubber]

Not much luck with attempt 2

Updated Dockerfile to allow child processes inherit the capabilities too and also assign capabilities also to iptables binary. Same output. Wonder if iptables 1.8.2 legacy on debian buster is good enough. i see mentions buster also now has nftables, successor to iptables.

[16:23] [ptolani@Pankajs-MacBook-Pro:~/greaterbank/infosec/PE-1247/pankajmt/linkerd2-proxy-init]$ [._.] git diff Dockerfile
diff --git a/Dockerfile b/Dockerfile
index 154d9bc..8a0d098 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -18,9 +18,16 @@ RUN apt-get update \
     && apt-get install -y --no-install-recommends \
         iptables \
         procps \
+        libcap2-bin \
     && rm -rf /var/lib/apt/lists/* \
     && update-alternatives --set iptables /usr/sbin/iptables-legacy \
     && update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
+
+RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
 COPY LICENSE /linkerd/LICENSE
 COPY --from=golang /out/linkerd2-proxy-init /usr/local/bin/proxy-init
+RUN setcap cap_net_raw,cap_net_admin+eip /usr/local/bin/proxy-init
+RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
+USER linkerd
 ENTRYPOINT ["/usr/local/bin/proxy-init"]

[part-time-githubber]

Looks like I am able to run iptables commands in a pod without being root if the associated PSP in addition to capabilities NET_ADMIN and NET_RAW also has allowPrivilegeEscalation: true

Now as per https://kubernetes.io/docs/concepts/policy/pod-security-policy/,

This bool directly controls whether the no_new_privs flag gets set on the container process. This flag will prevent setuid binaries from changing the effective user ID, and prevent files from enabling extra capabilities (e.g. it will prevent the use of the ping tool). This behavior is required to effectively enforce MustRunAsNonRoot.

So, my understanding is, although not root, with your own binary, you can do things like root by including this in a docker image with setuid bit set or additional capabilities. so equally risky 👎

[wmorgan]

@pankajmt Thanks for the experimentation. To confirm, the only way you've found to have the proxy-init container to run as non-root is by setting allowPrivilegeEscalation to true in the PSP?

[part-time-githubber]

Following linkerd control plane PSP changes allow iptables to run as NON root

<   allowPrivilegeEscalation: false
---
>   allowPrivilegeEscalation: true
29c19
<     rule: RunAsAny
---
>     rule: MustRunAsNonRoot

my docker image where I have this working atm is

FROM ubuntu

RUN apt-get update && apt-get install -y --no-install-recommends iptables libcap2-bin

RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
RUN touch /run/xtables.lock && chmod 0666 /run/xtables.lock

USER linkerd

I am expecting the same result for proxy init container, but still to be verified.

But I am now also wondering if the powers are still same as before.

Run as Root with limited capabilities vs Run as Not Root, allow privilege escalation and same limited capabilities

My experience with linux dates from setuid days, then there was a gap when all the capabilities model evolved. And I have been an applications person all along, never a OS geek :-)

Plan to read more.

[xynova]

Following linkerd control plane PSP changes allow iptables to run as NON root

<   allowPrivilegeEscalation: false
---
>   allowPrivilegeEscalation: true
29c19
<     rule: RunAsAny
---
>     rule: MustRunAsNonRoot

my docker image where I have this working atm is

FROM ubuntu

RUN apt-get update && apt-get install -y --no-install-recommends iptables libcap2-bin

RUN groupadd -r linkerd && useradd -r -g linkerd linkerd
RUN setcap cap_net_raw,cap_net_admin+eip /usr/sbin/xtables-legacy-multi
RUN touch /run/xtables.lock && chmod 0666 /run/xtables.lock

USER linkerd

I am expecting the same result for proxy init container, but still to be verified.

But I am now also wondering if the powers are still same as before.

Run as Root with limited capabilities vs Run as Not Root, allow privilege escalation and same limited capabilities

My experience with linux dates from setuid days, then there was a gap when all the capabilities model evolved. And I have been an applications person all along, never a OS geek :-)

Plan to read more.

Running as non-root seems to be always a better choice. Granting root means the process gets write access to all files though and no capabilities prevent that from my experiments.

The allow privilege escalation is equivalent to docker run --security-opt=no-new-privileges. I am not entirely sure how the capabilites workflow happens in linux but it seems that granting capabilities does not mean that the process automatically has them when run. I have the feeling it still has to request them once trying to perform a privilege action and then this no-new-privileges prevents the process from getting anything on top of what it started with.

[chrischdi]

The changes at https://github.com/linkerd/linkerd2-proxy-init/pull/49 lead to an up and running linkerd without the need of allowPrivilegeEscalation: true. The trick is to add the capabilities to the proxi-init binary too, because if it doesn't have them it cannot call a binary which requires them.

However, for now I am stuck at the injected init containers which are currently not allowed to run :-)

[chrischdi]

Requires for me the following changes in this repo for the controller:

diff --git a/charts/partials/templates/_proxy-init.tpl b/charts/partials/templates/_proxy-init.tpl
index 32411552..93e23a37 100644
--- a/charts/partials/templates/_proxy-init.tpl
+++ b/charts/partials/templates/_proxy-init.tpl
@@ -44,8 +44,6 @@ securityContext:
   privileged: false
   {{- end }}
   readOnlyRootFilesystem: true
-  runAsNonRoot: false
-  runAsUser: 0
 terminationMessagePolicy: FallbackToLogsOnError
 {{- if or (not .Values.cniEnabled) .Values.proxyInit.saMountPath }}
 volumeMounts:

And so I'm running completely as non-root :-)