Open ohault opened 6 months ago
Hi @ohault
What environment are you using? Is this perhaps on LXD? One thing you could try as a work-around is to edit the multus daemonset and remove the MountPropagation: Bidirectional
line from https://github.com/canonical/microk8s-community-addons/blob/ec3dddc5a3b7efac45fe81daca35eba6b3b81ec0/addons/multus/multus.yaml#L160
I'm using WSL2.
Hi @ohault
What environment are you using? Is this perhaps on LXD? One thing you could try as a work-around is to edit the multus daemonset and remove the
MountPropagation: Bidirectional
line from https://github.com/canonical/microk8s-community-addons/blob/ec3dddc5a3b7efac45fe81daca35eba6b3b81ec0/addons/multus/multus.yaml#L160
I'm using WSL2.
OK, thank you. Could you check whether the proposed workaround above helps?
OK, thank you. Could you check whether the proposed workaround above helps?
No so easy to test as multus.yaml is in ro in the /snap tree because I used snap to install microk8s. Feel free to you tell me how I can test it ? I can do it under WSL2/Ubuntu 20.4.6.
OK, the addons themselves are read-write, and can be found under /var/snap/microk8s/common/addons
. You could edit the manifest at /var/snap/microk8s/common/addons/community/addons/multus
then enable the addon
OK, the addons themselves are read-write, and can be found under
/var/snap/microk8s/common/addons
. You could edit the manifest at/var/snap/microk8s/common/addons/community/addons/multus
then enable the addon
It's exactly what I tried, but I got an error message like "The filesystem is read only".
I think there must be a mixup here. The read-only parts of microk8s will be in /snap/microk8s/current/...
. /var/snap/microk8s/common/...
will not be readonly (it is actively used for microk8s to store and maintain the current state).
Can you try editing the file again? The command should be
sudo vim /var/snap/microk8s/common/addons/community/addons/multus/multus.yaml
Or can you share the exact error message that you are getting?
I have just tried from a fresh new system. I confirm that on the new system, I have been able to modify the file /var/snap/microk8s/common/addons/community/addons/multus/multus.yaml I don't understand why this was not the case, but it's a detail. I confirm also that by removing the line "MountPropagation: Bidirectional" fix the multus starting issue.
Summary
Using stable channel 1.28/stable, multus daemonset cannot be started meanwhile it can be started using 1.24/stable
Reproduction Steps
sudo snap install microk8s --classic microk8s enable multus => Waiting for multus daemonset to start................................
sudo snap install microk8s --classic --channel=1.24/stable microk8s enable multus => Waiting for multus daemonset to start................................ Multus is enabled Multus is enabled with version: multus-cni version:v3.4.2, commit:4eac660359f223d34bcaf0fddbc42fd542f02ba1, date:2020-05-15T12:43:46+0000
Introspection Report
microk8s kubectl describe pod -n kube-system
=>
Name: kube-multus-ds-9mnfc Namespace: kube-system Priority: 0 Service Account: multus Node: pcwin11oha/172.27.218.10 Start Time: Sun, 24 Mar 2024 12:05:10 +0100 Labels: app=multus controller-revision-hash=69c976674c name=multus pod-template-generation=1 tier=node Annotations:
Status: Pending
IP: 172.27.218.10
IPs:
IP: 172.27.218.10
Controlled By: DaemonSet/kube-multus-ds
Init Containers:
install-multus-binary:
Container ID:
Image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9
Image ID:
Port:
Host Port:
Command:
cp
/usr/src/multus-cni/bin/multus
/host/opt/cni/bin/multus
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Requests:
cpu: 10m
memory: 15Mi
Environment:
Mounts:
/host/opt/cni/bin from cnibin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h56gq (ro)
Containers:
kube-multus:
Container ID:
Image: ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9
Image ID:
Port:
Host Port:
Command:
/entrypoint.sh
Args:
--multus-conf-file=auto
--multus-kubeconfig-file-host=/var/snap/microk8s/current/args/cni-network/multus.d/multus.kubeconfig
--cni-version=0.3.1
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
Mounts:
/host/etc/cni/net.d from cni (rw)
/host/opt/cni/bin from cnibin (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h56gq (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
cni:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/args/cni-network/
HostPathType:
cnibin:
Type: HostPath (bare host directory volume)
Path: /var/snap/microk8s/current/opt/cni/bin/
HostPathType:
kube-api-access-h56gq:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: Burstable
Node-Selectors:
Tolerations: :NoSchedule op=Exists
node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
Normal Scheduled 5m7s default-scheduler Successfully assigned kube-system/kube-multus-ds-9mnfc to pcwin11oha Normal Pulling 5m7s kubelet Pulling image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9" Normal Pulled 4m34s kubelet Successfully pulled image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9" in 32.058s (32.058s including waiting) Warning Failed 4m34s kubelet Error: failed to generate container "50566bcd392f86d00a35d4aded6520f1e32dc0f373ed68ec90aed66cf175a8e8" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 4m34s kubelet Error: failed to generate container "93dfaf569e709eb49cdb7219cbab9fa21043e8022aeb2648f738939c0220247a" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 4m21s kubelet Error: failed to generate container "c4c5f7b86be8da6eaf7f881a3de956b5f03c61984d28ae99feb4d09b8be73dab" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 4m8s kubelet Error: failed to generate container "0b1ecb344511f725ce7a0ad7fc8dad952d13aa63da1c852dcd96e2f1febc9984" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 3m53s kubelet Error: failed to generate container "717c88b449c5ff933b7a53f5f460339ddb382abf7d6170cdcd2d53a73ba927f1" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 3m41s kubelet Error: failed to generate container "8fc8635ed7ea0967fdab20745e0811ee8125a7b082037188cf4af56912dc6683" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 3m29s kubelet Error: failed to generate container "6541ec4495b46a00116b3a7af15fb554a2e0bb1dd478d64b10c87daf041c1418" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 3m14s kubelet Error: failed to generate container "0252bd0225b048d393b671fd46439c28a3b315ab5c8dc71702983f6944036b75" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 3m3s kubelet Error: failed to generate container "599cb58f63f08c8065aca17f902d908b20c1a846951b91177e95429894062b6b" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Warning Failed 2m23s (x3 over 2m49s) kubelet (combined from similar events): Error: failed to generate container "5cc95b135d7b09cbf3cedd6eb9f061a1aeb827cadc4706914dc65301f048b587" spec: failed to generate spec: path "/var/snap/microk8s/current/opt/cni/bin/" is mounted on "/" but it is not a shared mount Normal Pulled 6s (x22 over 4m34s) kubelet Container image "ghcr.io/k8snetworkplumbingwg/multus-cni:v3.9" already present on machine