Closed johnmuth81 closed 1 year ago
johnmuth81
commented
2 years ago I found an issue on the cri-o github from Aug 2020 that looks amazingly similar: https://github.com/cri-o/cri-o/issues/3555. They closed it with a "disable ipv6" workaround.
jszwedko
commented
2 years ago We ran into this today when upgrading minikube for https://github.com/vectordotdev/vector . We tried disabling IPv6 from the bridge as mentioned in https://github.com/kubernetes/minikube/issues/12928#issuecomment-966508095 . That did get rid of the iptables error shown in the logs in the linked issue, but we still see the same error for kube-proxy:
"container create failed: time=\"2021-12-27T18:11:24Z\" level=error msg=\"container_linux.go:380: starting container process caused: apply caps: operation not permitted\"\n",Some more version information:
minikube kubectl -- version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:41:28Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"clean", BuildDate:"2021-10-27T18:35:25Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}Linux minikube 5.10.47-linuxkit #1 SMP Sat Jul 3 21:51:47 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxIt did actually launch correctly on another machine I have which has an older kernel:
Linux a-8cktst39ij0a 4.14.232-176.381.amzn2.x86_64 #1 SMP Wed May 19 00:31:54 UTC 2021 x86_64 x86_64 x86_64 GNU/LinuxSo it may be kernel version related.
klaases
commented
2 years ago Hi @johnmuth81, were you able to disable ipv6 and did @jszwedko's comment help resolve this at least temporarily?
A new release is on its way, which may help to further resolve this issue.
spowelljr
commented
2 years ago Hi @johnmuth81, we haven't heard back from you, do you still have this issue? There isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.
I will close this issue for now but feel free to reopen when you feel ready to provide more information.
jszwedko
commented
2 years ago Hi @spowelljr !
This is still an issue for us with minikube 1.25.1. We are using Ubuntu 20.04. To reproduce I:
ami-0a93ee71d4e382474 in us-east-2)ubuntu user to the docker group, and logged out and back inI then ran the command in the original issue:
minikube start --container-runtime=cri-o --wait=allAnd got the same effect, kube-proxy doesn't start:
ubuntu@ip-172-31-40-215:~$ minikube kubectl -- get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-64897985d-9wsvs 0/1 ContainerCreating 0 25s
kube-system etcd-minikube 1/1 Running 0 40s
kube-system kindnet-p4r5t 1/1 Running 0 26s
kube-system kube-apiserver-minikube 1/1 Running 0 33s
kube-system kube-controller-manager-minikube 1/1 Running 0 33s
kube-system kube-proxy-hxgcj 0/1 CreateContainerError 0 26s
kube-system kube-scheduler-minikube 1/1 Running 0 33s
kube-system storage-provisioner 1/1 Running 0 24s
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"creationTimestamp": "2022-02-09T20:08:46Z",
"generateName": "kube-proxy-",
"labels": {
"controller-revision-hash": "8485885f8b",
"k8s-app": "kube-proxy",
"pod-template-generation": "1"
},
"name": "kube-proxy-hxgcj",
"namespace": "kube-system",
"ownerReferences": [
{
"apiVersion": "apps/v1",
"blockOwnerDeletion": true,
"controller": true,
"kind": "DaemonSet",
"name": "kube-proxy",
"uid": "3f55e718-c057-4714-bc7c-63998b8e1606"
}
],
"resourceVersion": "533",
"uid": "dd80f370-7a55-4830-8daf-e7af6c90344b"
},
"spec": {
"affinity": {
"nodeAffinity": {
"requiredDuringSchedulingIgnoredDuringExecution": {
"nodeSelectorTerms": [
{
"matchFields": [
{
"key": "metadata.name",
"operator": "In",
"values": [
"minikube"
]
}
]
}
]
}
}
},
"containers": [
{
"command": [
"/usr/local/bin/kube-proxy",
"--config=/var/lib/kube-proxy/config.conf",
"--hostname-override=$(NODE_NAME)"
],
"env": [
{
"name": "NODE_NAME",
"valueFrom": {
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "spec.nodeName"
}
}
}
],
"image": "k8s.gcr.io/kube-proxy:v1.23.1",
"imagePullPolicy": "IfNotPresent",
"name": "kube-proxy",
"resources": {},
"securityContext": {
"privileged": true
},
"terminationMessagePath": "/dev/termination-log",
"terminationMessagePolicy": "File",
"volumeMounts": [
{
"mountPath": "/var/lib/kube-proxy",
"name": "kube-proxy"
},
{
"mountPath": "/run/xtables.lock",
"name": "xtables-lock"
},
{
"mountPath": "/lib/modules",
"name": "lib-modules",
"readOnly": true
},
{
"mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
"name": "kube-api-access-lv652",
"readOnly": true
}
]
}
],
"dnsPolicy": "ClusterFirst",
"enableServiceLinks": true,
"hostNetwork": true,
"nodeName": "minikube",
"nodeSelector": {
"kubernetes.io/os": "linux"
},
"preemptionPolicy": "PreemptLowerPriority",
"priority": 2000001000,
"priorityClassName": "system-node-critical",
"restartPolicy": "Always",
"schedulerName": "default-scheduler",
"securityContext": {},
"serviceAccount": "kube-proxy",
"serviceAccountName": "kube-proxy",
"terminationGracePeriodSeconds": 30,
"tolerations": [
{
"operator": "Exists"
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/not-ready",
"operator": "Exists"
},
{
"effect": "NoExecute",
"key": "node.kubernetes.io/unreachable",
"operator": "Exists"
},
{
"effect": "NoSchedule",
"key": "node.kubernetes.io/disk-pressure",
"operator": "Exists"
},
{
"effect": "NoSchedule",
"key": "node.kubernetes.io/memory-pressure",
"operator": "Exists"
},
{
"effect": "NoSchedule",
"key": "node.kubernetes.io/pid-pressure",
"operator": "Exists"
},
{
"effect": "NoSchedule",
"key": "node.kubernetes.io/unschedulable",
"operator": "Exists"
},
{
"effect": "NoSchedule",
"key": "node.kubernetes.io/network-unavailable",
"operator": "Exists"
}
],
"volumes": [
{
"configMap": {
"defaultMode": 420,
"name": "kube-proxy"
},
"name": "kube-proxy"
},
{
"hostPath": {
"path": "/run/xtables.lock",
"type": "FileOrCreate"
},
"name": "xtables-lock"
},
{
"hostPath": {
"path": "/lib/modules",
"type": ""
},
"name": "lib-modules"
},
{
"name": "kube-api-access-lv652",
"projected": {
"defaultMode": 420,
"sources": [
{
"serviceAccountToken": {
"expirationSeconds": 3607,
"path": "token"
}
},
{
"configMap": {
"items": [
{
"key": "ca.crt",
"path": "ca.crt"
}
],
"name": "kube-root-ca.crt"
}
},
{
"downwardAPI": {
"items": [
{
"fieldRef": {
"apiVersion": "v1",
"fieldPath": "metadata.namespace"
},
"path": "namespace"
}
]
}
}
]
}
}
]
},
"status": {
"conditions": [
{
"lastProbeTime": null,
"lastTransitionTime": "2022-02-09T20:08:46Z",
"status": "True",
"type": "Initialized"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-02-09T20:08:46Z",
"message": "containers with unready status: [kube-proxy]",
"reason": "ContainersNotReady",
"status": "False",
"type": "Ready"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-02-09T20:08:46Z",
"message": "containers with unready status: [kube-proxy]",
"reason": "ContainersNotReady",
"status": "False",
"type": "ContainersReady"
},
{
"lastProbeTime": null,
"lastTransitionTime": "2022-02-09T20:08:46Z",
"status": "True",
"type": "PodScheduled"
}
],
"containerStatuses": [
{
"image": "k8s.gcr.io/kube-proxy:v1.23.1",
"imageID": "",
"lastState": {},
"name": "kube-proxy",
"ready": false,
"restartCount": 0,
"started": false,
"state": {
"waiting": {
"message": "container create failed: time=\"2022-02-09T20:09:58Z\" level=error msg=\"container_linux.go:380: starting container process caused: apply caps: operation not permitted\"\n",
"reason": "CreateContainerError"
}
}
}
],
"hostIP": "192.168.49.2",
"phase": "Pending",
"podIP": "192.168.49.2",
"podIPs": [
{
"ip": "192.168.49.2"
}
],
"qosClass": "BestEffort",
"startTime": "2022-02-09T20:08:46Z"
}
}Additional version information:
ubuntu@ip-172-31-40-215:~$ minikube version
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d
ubuntu@ip-172-31-40-215:~$ docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Built: Mon Dec 13 11:45:33 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.12
Git commit: 459d0df
Built: Mon Dec 13 11:43:42 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.12
GitCommit: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
runc:
Version: 1.0.2
GitCommit: v1.0.2-0-g52b36a2
docker-init:
Version: 0.19.0
GitCommit: de40ad0Is there any more information that would be useful?
spowelljr
commented
2 years ago Hi @jszwedko, your kube-proxy error seems to be starting container process caused: apply caps: operation not permitted.
This seems related to LXC itself, here's a related issue: https://serverfault.com/q/946854
jszwedko
commented
2 years ago Ah, I see. Thanks for the response and the link! I'm still not completely following though, how would I apply the recommendations there to minikube? My same system is able to run docker run without issue (which is different from the original post on https://serverfault.com/q/946854). It also only fails with the cri-o runtime (docker and containerd work fine).
mqasimsarfraz
commented
2 years ago Hi all, I deep dived into the issue and found out the root cause is a missing capabilities in the minikube docker container. cri-o updated the capability list and if the container in which cri-o will eventually be launched doesn't have those capabilities in CapBnd we are expected to get operation not permitted while applying those additional capabilities. You can do a quick test by running following and if it works fine you are good:
docker run --rm --cap-add CAP_BPF hello-worldOr you can check the capabilities in the container by:
root@minikube:/# cat /proc/self/status | grep ^Cap
CapInh: 0000000000000000
CapPrm: 0000003fffffffff
CapEff: 0000003fffffffff
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
root@minikube:/# capsh --decode=0000003fffffffff
WARNING: libcap needs an update (cap=40 should have a name).
0x0000003fffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_readwe can see CAP_PERFMON, CAP_BPF, and CAP_CHECKPOINT_RESTORE are missing from the list. This was on the host running docker 20.10.18 and I figured out it was missing the fix. The fix is already merged to master so upgrading to docker 22.06.0-beta.0 solves the issue:
root@minikube:/# cat /proc/self/status | grep ^Cap
CapInh: 0000000000000000
CapPrm: 000001ffffffffff
CapEff: 000001ffffffffff
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
root@minikube:/# capsh --decode=000001ffffffffff
WARNING: libcap needs an update (cap=40 should have a name).
0x000001ffffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,38,39,40Currently, I see running a test docker release as an only workaround. I will add comment to the fix that it should be back ported to stable release.
mqasimsarfraz
commented
2 years ago @spowelljr any thoughts on this?
josepdcs
commented
1 year ago I'm experiencing the same error right now by running:
minikube -p minikube-crio --container-runtime=cri-o --driver=docker start
The pod/kube-proxy-vxrrm still with status CreateContainerError
kubectl describe pod/kube-proxy-vxrrm --namespace kube-syste
spowelljr
commented
1 year ago @spowelljr any thoughts on this?
Sorry for the delay, thanks for the deep dive on the issue, it makes complete sense why it's not working. We should warn users about this when they try to start minikube with docker and cri-o and link them to this issue. Unfortunately it doesn't seem like there's anything else we can do from our side. Once Docker makes the version with the fix GA we can update the message to suggest users to update to the latest Docker version.
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle stale/lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/remove-lifecycle rotten/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
k8s-triage-robot
commented
1 year ago The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages issues according to the following rules:
lifecycle/stale is appliedlifecycle/stale was applied, lifecycle/rotten is appliedlifecycle/rotten was applied, the issue is closedYou can:
/reopen/remove-lifecycle rottenPlease send feedback to sig-contributor-experience at kubernetes/community.
/close not-planned
k8s-ci-robot
commented
1 year ago @k8s-triage-robot: Closing this issue, marking it as "Not Planned".
Minikube Version: v1.24.0 System: Linux. Ubuntu 20.04, Fedora 33
Running
kubectl get pod -Afrom another terminal:Output from
minikube logs: