search

canonical / microk8s

MicroK8s is a small, fast, single-package Kubernetes for datacenters and the edge.
https://microk8s.io
Apache License 2.0
8.47k stars 772 forks source link

microks8 flannel - subnet.env not found #935

Closed LuffyyyffuL closed 2 years ago

LuffyyyffuL commented 4 years ago

Hello,

My server :

Ubuntu 19.10 microk8s 1.16

I have a problem with microk8s and flannel.

When a run this command :

microk8s.kubectl -n kube-system describe pod kubernetes-dashboard-5c848cc544-dnq66

I have this log :

Warning FailedCreatePodSandBox 15m kubelet, monkeydryzen Failed create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a5f4e184e1fd364b99dee83b32715ca63e3acba3dc6c3df1d338c15f1be3f126": open /var/snap/microk8s/common/run/flannel/subnet.env: no such file or directory

"flannel" folder do not exist "subnet.env" file do not exist

When a run this command :

microk8s.kubectl get all --all-namespaces

I have this log :

`NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/dashboard-metrics-scraper-687667bb6c-4cbwb 0/1 ContainerCreating 0 20m kube-system pod/heapster-v1.5.2-5c58f64f8b-48sww 0/4 ContainerCreating 0 20m kube-system pod/kubernetes-dashboard-5c848cc544-dnq66 0/1 ContainerCreating 0 20m kube-system pod/monitoring-influxdb-grafana-v4-6d599df6bf-5vkmg 0/2 ContainerCreating 0 20m

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.152.183.1 443/TCP 26m kube-system service/dashboard-metrics-scraper ClusterIP 10.152.183.161 8000/TCP 20m kube-system service/heapster ClusterIP 10.152.183.233 80/TCP 20m kube-system service/kubernetes-dashboard ClusterIP 10.152.183.224 443/TCP 20m kube-system service/monitoring-grafana ClusterIP 10.152.183.205 80/TCP 20m kube-system service/monitoring-influxdb ClusterIP 10.152.183.246 8083/TCP,8086/TCP 20m

NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system deployment.apps/dashboard-metrics-scraper 0/1 1 0 20m kube-system deployment.apps/heapster-v1.5.2 0/1 1 0 20m kube-system deployment.apps/kubernetes-dashboard 0/1 1 0 20m kube-system deployment.apps/monitoring-influxdb-grafana-v4 0/1 1 0 20m

NAMESPACE NAME DESIRED CURRENT READY AGE kube-system replicaset.apps/dashboard-metrics-scraper-687667bb6c 1 1 0 20m kube-system replicaset.apps/heapster-v1.5.2-5c58f64f8b 1 1 0 20m kube-system replicaset.apps/kubernetes-dashboard-5c848cc544 1 1 0 20m kube-system replicaset.apps/monitoring-influxdb-grafana-v4-6d599df6bf 1 1 0 20m`

Have you a idea ?

Thank you

balchua commented 4 years ago

@LuffyyyffuL will it be possible to attach the inpect tarball? You can produce that by executing microk8s.inspect.

If not, maybe you can check flannel logs by, sudo journalctl -u snap.microk8s.daemon-flanneld -f

LuffyyyffuL commented 4 years ago

Hi,

microk8s.inspect :

Inspecting services Service snap.microk8s.daemon-cluster-agent is running Service snap.microk8s.daemon-flanneld is running Service snap.microk8s.daemon-containerd is running Service snap.microk8s.daemon-apiserver is running Service snap.microk8s.daemon-apiserver-kicker is running Service snap.microk8s.daemon-proxy is running Service snap.microk8s.daemon-kubelet is running Service snap.microk8s.daemon-scheduler is running Service snap.microk8s.daemon-controller-manager is running Service snap.microk8s.daemon-etcd is running Copy service arguments to the final report tarball Inspecting AppArmor configuration Gathering system information Copy processes list to the final report tarball Copy snap list to the final report tarball Copy VM name (or none) to the final report tarball Copy disk usage information to the final report tarball Copy memory usage information to the final report tarball Copy server uptime to the final report tarball Copy current linux distribution to the final report tarball Copy openSSL information to the final report tarball Copy network configuration to the final report tarball Inspecting kubernetes cluster Inspect kubernetes cluster

Thank you

balchua commented 4 years ago

It will generate a tarball and if possible upload it here?

LuffyyyffuL commented 4 years ago

I have begining to read, there are differents errors, but i share whith you :

https://drive.google.com/file/d/1ke-KCZjr9KkkVUiDPbDH7kbwZmTqClli/view?usp=sharing

In snap.microk8s.daemon-flanneld journal.log, I have this error :

microk8s.daemon-flanneld[2268]: E0125 12:31:45.368281 2268 watch.go:43] Watch subnets: client: etcd cluster is unavailable or misconfigured; error #0: unexpected EOF

and

microk8s.daemon-flanneld[23642]: E0126 13:37:56.058580 23642 main.go:289] Error registering network: failed to configure interface flannel.1: failed to ensure address of interface flannel.1: link has incompatible addresses. Remove additional addresses and try again. &netlink.Vxlan{LinkAttrs:netlink.LinkAttrs{Index:4, MTU:1450, TxQLen:0, Name:"flannel.1", HardwareAddr:net.HardwareAddr{0xfa, 0xda, 0xda, 0xe1, 0x22, 0x4d}, Flags:0x13, RawFlags:0x11043, ParentIndex:0, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0xc4206e5104), Promisc:0, Xdp:(*netlink.LinkXdp)(0xc420287520), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x0}, VxlanId:1, VtepDevIndex:2, SrcAddr:net.IP{0xc0, 0xa8, 0x0, 0xb}, Group:net.IP(nil), TTL:0, TOS:0, Learning:false, Proxy:false, RSC:false, L2miss:false, L3miss:false, UDPCSum:true, NoAge:false, GBP:false, Age:300, Limit:0, Port:8472, PortLow:0, PortHigh:0}

For information, on my server i have :

OpenVPN running on the 10.8.1.0 network PiHole Ufw

balchua commented 4 years ago

I have seen such error in flannel logs.

Error registering network: failed to configure interface flannel.1: failed to ensure address of interface flannel.1: link has incompatible addresses. Remove additional addresses and try again.

Can you try to stop microk8s, delete flannel interface. microk8s.stop then sudo ip link delete flannel.1 Finally microk8s.start

LuffyyyffuL commented 4 years ago

Hi, thank for your reply

I have delete interface flannel and i have reinstate microk8s.

Now i have a other problem, when i run "ip show", i can see news interfaces "veth" come and go.

jonasscherer commented 4 years ago

I have the same issue with: microk8s v1.18 Ubuntu 18.04

This issue emerged after the last microk8s update. Before that everything was working great..

Edit:

when I manually create /var/snap/microk8s/common/run/flannel/subnet.env:

FLANNEL_NETWORK=10.1.0.0/16
FLANNEL_SUBNET=10.1.28.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=false

Everything is working again. Any idea, why it's not created automatically?

trulede commented 4 years ago

when I manually create /var/snap/microk8s/common/run/flannel/subnet.env:

I encountered this problem too, after a reinstall of Microk8s ... creating that file helped.

APoole00 commented 4 years ago

I am also getting the same issue.

gaofeng1973 commented 4 years ago

I am also getting the same issue both with --channel=latest/stable and --channel=1.18/stable

ktsakalozos commented 4 years ago

@gaofeng1973 could you please attach the microk8s.inspect tarball?

gaofeng1973 commented 4 years ago

@gaofeng1973 could you please attach the microk8s.inspect tarball?

@LuffyyyffuL will it be possible to attach the inpect tarball? You can produce that by executing microk8s.inspect.

I meet the same question but I found this issue was caused by the host has run some squid proxy.

my environment:
ubunut20.04  IP:192.168.0.59
sudo snap install microk8s --classic --channel=1.18/stable

my /etc/environment setting is :

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
http_proxy=http://xxxxxx:10080
https_proxy=http://xxxxxx:10080
no_proxy=192.168.0.59,192.168.0.66,localhost,127.0.0.1

when no_proxy variable set without localhost,127.0.0.1, the /var/snap/microk8s/common/run/flannel/subnet.env missing issue will happened. otherwise this issue was disappeared, so I think maybe this issue caused by environment variable no_proxy setting where some squid proxy running in this node.

Could this help you to verify and reproduce this issue?

donbowman commented 3 years ago

Also observing this problem. For me the culprit is the flannel service crasing w/ sigsegv. microk8s.inspect attached inspection-report-20201208_134629.tar.gz

uname -a
Linux cube 5.10.0-051000rc6-generic #202011291930 SMP Mon Nov 30 00:36:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.10
DISTRIB_CODENAME=groovy
DISTRIB_DESCRIPTION="Ubuntu 20.10"
microk8s                 v1.18.9 (via snap)
cat /var/snap/microk8s/1702/args/flanneld 
--iface=""
--etcd-endpoints=https://127.0.0.1:12379
--etcd-cafile=${SNAP_DATA}/certs/ca.crt
--etcd-certfile=${SNAP_DATA}/certs/server.crt
--etcd-keyfile=${SNAP_DATA}/certs/server.key
--subnet-file=${SNAP_COMMON}/run/flannel/subnet.env

It may match https://github.com/coreos/flannel/issues/1365

Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691638]: PrevNode.Value: {"Network": "10.1.0.0/16", "Backend": {"Type": "vxlan"}}
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691645]: {"Network": "10.1.0.0/16", "Backend": {"Type": "vxlan"}}
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: fatal error: unexpected signal during runtime execution
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: [signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0x7fde247df96e]
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime stack:
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime.throw(0x19a3a14, 0x2a)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/panic.go:616 +0x81
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime.sigpanic()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/signal_unix.go:372 +0x28e
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: goroutine 1 [syscall, locked to thread]:
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime.cgocall(0x14432e0, 0xc420095c50, 0x19a184f)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/cgocall.go:128 +0x64 fp=0xc420095c08 sp=0xc420095bd0 pc=0x402304
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user._Cfunc_mygetpwuid_r(0x0, 0xc4200c4d20, 0x45b9460, 0x400, 0xc42000e048, 0x0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         _cgo_gotypes.go:170 +0x4d fp=0xc420095c50 sp=0xc420095c08 pc=0x6c637d
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.lookupUnixUid.func1.1(0x0, 0xc4200c4d20, 0x45b9460, 0x400, 0xc42000e048, 0xc420095ce0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x13f fp=0xc420095c90 sp=0xc420095c50 pc=0x6c79ef
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.lookupUnixUid.func1(0xc420095cb0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/cgo_lookup_unix.go:100 +0x50 fp=0xc420095cd0 sp=0xc420095c90 pc=0x6c7a90
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.retryWithBuffer(0xc420086a70, 0xc420095dc0, 0xc420086a70, 0x0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/cgo_lookup_unix.go:253 +0x3b fp=0xc420095d30 sp=0xc420095cd0 pc=0x6c6ebb
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.lookupUnixUid(0x0, 0x0, 0x0, 0x0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x130 fp=0xc420095df8 sp=0xc420095d30 pc=0x6c6800
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.current(0xc420095e60, 0x10, 0xc420095e58)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/cgo_lookup_unix.go:49 +0x27 fp=0xc420095e28 sp=0xc420095df8 pc=0x6c6697
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.Current.func1()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/lookup.go:11 +0x22 fp=0xc420095e50 sp=0xc420095e28 pc=0x6c7832
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: sync.(*Once).Do(0x27a14c0, 0x19f7218)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/sync/once.go:44 +0xbe fp=0xc420095e88 sp=0xc420095e50 pc=0x46c45e
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/user.Current(0xc42003846c, 0x4, 0xc42003846c)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/user/lookup.go:11 +0x3d fp=0xc420095eb8 sp=0xc420095e88 pc=0x6c5f5d
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: github.com/coreos/flannel/vendor/github.com/golang/glog.init.1()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /go/src/github.com/coreos/flannel/vendor/github.com/golang/glog/glog_file.go:63 +0x39 fp=0xc420095f10 sp=0xc420095eb8 pc=0x6cc4f9
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: github.com/coreos/flannel/vendor/github.com/golang/glog.init()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         <autogenerated>:1 +0x1c2 fp=0xc420095f58 sp=0xc420095f10 pc=0x6cd212
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: main.init()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         <autogenerated>:1 +0x87 fp=0xc420095f88 sp=0xc420095f58 pc=0x1441b87
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime.main()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/proc.go:186 +0x1ca fp=0xc420095fe0 sp=0xc420095f88 pc=0x42be9a
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: runtime.goexit()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420095fe8 sp=0xc420095fe0 pc=0x456f71
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: goroutine 5 [syscall]:
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/signal.signal_recv(0x0)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/runtime/sigqueue.go:139 +0xa6
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: os/signal.loop()
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: created by os/signal.init.0
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: goroutine 6 [chan receive]:
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: github.com/coreos/flannel/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x27a1f40)
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /go/src/github.com/coreos/flannel/vendor/github.com/golang/glog/glog.go:879 +0x8b
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]: created by github.com/coreos/flannel/vendor/github.com/golang/glog.init.0
Dec 08 13:43:28 cube microk8s.daemon-flanneld[2691559]:         /go/src/github.com/coreos/flannel/vendor/github.com/golang/glog/glog.go:410 +0x203
Dec 08 13:43:28 cube systemd[1]: snap.microk8s.daemon-flanneld.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Dec 08 13:43:28 cube systemd[1]: snap.microk8s.daemon-flanneld.service: Failed with result 'exit-code'.
Dec 08 13:43:28 cube systemd[1]: snap.microk8s.daemon-flanneld.service: Scheduled restart job, restart counter is at 45.
donbowman commented 3 years ago

OK i have confirmed, https://github.com/coreos/flannel/issues/1365 is the same issue. flanneld crashes on my system.

the fix is to compile flannel w/ golang v1.15+

$ curl -sSL https://github.com/coreos/flannel/releases/download/v0.13.0/flannel-v0.13.0-linux-amd64.tar.gz | tar Cxzv /tmp/ flanneld
flanneld
$ /tmp/flanneld -version
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe0 pc=0x7ffb199db96e]
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.