ErmakovDmitriy
commented
2 years ago Some steps to troubleshoot
In order to further test and check I deployed a new OKD 4.10 cluster with only 3 master nodes (I have quite limited resources) using Openshift SDN for network (the cluster in the bug used Openshift OVN-Kubernetes).
The CNI was redeployed with logLevel: debug. I could not see any logs in journalctl -u kubelet | grep linkerd from the CNI.
Probably, for some reason, the CNI plugin even is not called by Openshift (Multus). I deleted the linkerd-cni binary on one of the nodes and Pods still were normally started on it.
To further check and compare, I deployed Maistra Operator (Istio based) on the same cluster. It works well out-of-the-box. Comparing the Istio-CNI and Linkerd-CNI Pod configurations, I noticed that Istio uses:
# -- Directory on the host where the CNI plugin binaries reside
destCNINetDir: "/etc/cni/multus/net.d/"
# -- Directory on the host where the CNI configuration will be placed
destCNIBinDir: "/opt/multus/bin/"while the article about Linkerd (see the bug description) recommends to use:
destCNIBinDir: /var/lib/cni/bin
destCNINetDir: /etc/kubernetes/cni/net.dKubelet in my Openshift is configured with default settings as below:
May 27 15:07:43 master0.okd.dev.local hyperkube[1442]: I0527 15:07:43.627811 1442 flags.go:64] FLAG: --cni-bin-dir="/opt/cni/bin"
May 27 15:07:43 master0.okd.dev.local hyperkube[1442]: I0527 15:07:43.627814 1442 flags.go:64] FLAG: --cni-cache-dir="/var/lib/cni/cache"
May 27 15:07:43 master0.okd.dev.local hyperkube[1442]: I0527 15:07:43.627817 1442 flags.go:64] FLAG: --cni-conf-dir="/etc/cni/net.d"Possible cause of the issue
I have checked the Maistra Operator and Istio documenation:
- https://istio.io/latest/docs/setup/platform-setup/openshift/#additional-requirements-for-the-application-namespace (about NetworkAttachDefinitions)
- https://istio.io/latest/docs/setup/additional-setup/cni/#hosted-kubernetes-settings (Istio has a special
chained: falsemode for Openshift)
It may be that Multus used by Openshift does not support CNI chaining so the Maistra uses NetworkAttachDefinitions roughly this way:
- A Namespace is assigned to be controlled by some of the Maistra Istio control plane
- A NetworkAttachDefinition for Istio CNI is created in the namespace
- Pods, annotated with Istio proxy inject annotation, are modified by mutating web hook to contain: proxy container and
k8s.v1.cni.cncf.io/networks: istio-net-attach-definition-nameannotation
this way, Multus applies the Istio CNI to the Pod.
I could not reproduce this with Linkerd despite trying to put necessary files (linkerd-cni.conf) in /etc/cni/multus/net.d/linkerd.conf and creating a NetworkAttachDefinition for Multus.
My "try-to-reproduce" configurations:
linkerd-cni.conf:
{
"cniVersion": "0.3.0",
"name": "linkerd-cni",
"type": "linkerd-cni",
"log_level": "debug",
"policy": {
"type": "k8s",
"k8s_api_root": "https://172.30.0.1:443",
"k8s_auth_token": "<token>"
},
"kubernetes": {
"kubeconfig": "/etc/cni/multus/net.d//ZZZ-linkerd-cni-kubeconfig"
},
"linkerd": {
"incoming-proxy-port": 4143,
"outgoing-proxy-port": 4140,
"proxy-uid": 2102,
"ports-to-redirect": [],
"inbound-ports-to-ignore": [
"4191",
"4190"
],
"simulate": false,
"use-wait-flag": true
}
}
NetworkAttachDefinition:
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
name: linkerd-cni
spec:
config: ''and one of the Linkerd Pods was annotated with k8s.v1.cni.cncf.io/networks: linkerd-cni.
This lead to errors on creating a Pod like below:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_linkerd-destination-696d48756d-p4wxs_linkerd_cfbd3a42-c6f1-4eb4-9cc1-b57a0bc95b8d_0(2661b3cd99b38bd18ca498929cf2be89db85a3b1f2e33e859fb63910ea1058a1): error adding pod linkerd_linkerd-destination-696d48756d-p4wxs to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [linkerd/linkerd-destination-696d48756d-p4wxs/cfbd3a42-c6f1-4eb4-9cc1-b57a0bc95b8d:linkerd-cni]: error adding container to network "linkerd-cni": unexpected end of JSON inputon a good side, I managed to "call" the CNI plugin.
stale[bot]
What is the issue?
I deployed Linkerd following an article https://buoyant.io/2022/01/20/running-linkerd-on-openshift-4/. The deployment itself works. Linkerd check returns several warnings and one error about linkerd-viz (I do not think that my issue is related to them).
All Linkerd and Linkerd-CNI Pods are running and ready.
When I try to deploy the example application (emojivoto), I get errors from ReplicaSets as below because the Openshift SCC prohibits creating a Pod having containers with arbitrary UID:
My expectation is that we should be able to run Pods with arbitrary UIDs and we should not use Openshift SCC like
anyuidor even more privileged.How can it be reproduced?
To reproduce the issue one should follow https://buoyant.io/2022/01/20/running-linkerd-on-openshift-4/ article. One need to download the example "emojivoto" application and edit it to annotate the emojivoto namespace with "proxy" inject annotations:
Important notice: Linkerd-VIZ helm chart was deployed with Openshift Security Context Constraint "anyuid" allowed for it and its metrics I can see in the VIZ dashboard.
Pods for the emojivoto use default (restricted) SCC and they can not use "proxy" UID 2102.
Logs, error output, etc
Deployment status of one of the emojivoto components:
output of
linkerd check -o shortLinkerd core checks
linkerd-existence
‼ cluster networks can be verified the following nodes do not expose a podCIDR:
List of servers
control-plane-version
‼ control plane and cli versions match control plane running stable-2.11.2 but cli running edge-22.5.2 see https://linkerd.io/2/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
‼ control plane proxies and cli versions match linkerd-destination-5ccc4764f4-hf6rt running stable-2.11.2 but cli running edge-22.5.2 see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints
Linkerd extensions checks
linkerd-viz
‼ viz extension proxies and cli versions match grafana-6496fff755-r7nwc running stable-2.11.2 but cli running edge-22.5.2 see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints
Status check results are √
Environment
Kubernetes version: v1.22.1-1839+b93fd35dd03051-dirty OKD version: 4.9.0-0.okd-2022-02-12-140851 using OVN-Kubernetes Linkerd: stable-2.11.2 installed via Helm chart
Possible solution
Maybe we should somehow allow the injected proxy to run with "any" UID which will be then assigned by Openshift (OKD) using its mechanism. This may improve security as the containers with proxy will not run with the same UID in a cluster and allow running workloads on OKD without giving containers/Pods too much access.
Additional context
No response
Would you like to work on fixing this bug?
No response