search

k3d-io / k3d

Little helper to run CNCF's k3s in Docker
https://k3d.io/
MIT License
5.33k stars 456 forks source link

[BUG] cluster fails to start, possibly related to BTRFS #729

Open TeddyDD opened 3 years ago

TeddyDD commented 3 years ago

What did you do

Create cluster on machine with BTRFS storage

What did you expect to happen

Working cluster

Screenshots or terminal output

k3d-k3s-default-server-0 logs `tail -n 300 < logs` ``` /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:343 +0x85 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:341 +0x11d goroutine 4278 [select]: github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run(0xc00192eea0) /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:162 +0x1d6 created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watcher).Watch /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:122 +0x145 goroutine 3948 [select]: context.propagateCancel.func1(0x4d58160, 0xc000a74a60, 0x4cf5ce0, 0xc0015ad080) /usr/local/go/src/context/context.go:279 +0xda created by context.propagateCancel /usr/local/go/src/context/context.go:278 +0x1fb goroutine 3943 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch.func1(0x178, 0x0, 0x0, 0x0, 0x0, 0xc0060a8f60, 0xc0060a8ea0, 0xc0007bc680) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:353 +0xef created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:342 +0x43d goroutine 4267 [select]: github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run(0xc002edcdd0) /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:162 +0x1d6 created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watcher).Watch /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:122 +0x145 goroutine 3901 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00315b980) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:69 +0xc8 created by github.com/rancher/k3s/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:60 +0x172 goroutine 4289 [chan receive]: github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc008271d40, 0xc001f4a6c0) /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0x34 created by github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:129 +0xa5 goroutine 5043 [chan receive]: github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run.func1() /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:772 +0x5d github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000111f60) /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x5f github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc00b723f60, 0x4c9d560, 0xc001b6cde0, 0x3efd701, 0xc009666600) /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xad github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000111f60, 0x3b9aca00, 0x0, 0x1, 0xc009666600) /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x98 github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...) /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*processorListener).run(0xc0019ec700) /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/shared_informer.go:771 +0x95 github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc001173a30, 0xc00a2480a0) /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x51 created by github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start /go/src/github.com/rancher/k3s/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x65 goroutine 4430 [chan receive]: github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run.func1(0xc008271d40, 0xc00095c750) /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:130 +0x34 created by github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache.(*controller).Run /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/tools/cache/controller.go:129 +0xa5 goroutine 3898 [chan receive]: github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue.(*Type).updateUnfinishedWorkLoop(0xc0037bff80) /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue/queue.go:198 +0xac created by github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue.newQueue /go/src/github.com/rancher/k3s/vendor/k8s.io/client-go/util/workqueue/queue.go:58 +0x135 goroutine 4611 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch.func1(0x178, 0x0, 0x0, 0x0, 0x0, 0xc006228060, 0xc00031bc80, 0xc0003fd480) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:353 +0xef created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:342 +0x43d goroutine 4405 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch.func1(0xc0037be9c0, 0xc0037bea20, 0xc0037bea01, 0xc000ad7290, 0x2a) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:343 +0x85 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:341 +0x11d goroutine 4308 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch.func1(0x178, 0x0, 0x0, 0x0, 0x0, 0xc0037be660, 0xc0037be5a0, 0xc003f34a60) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:353 +0xef created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:342 +0x43d goroutine 4427 [select]: github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc001454f70, 0xc00287eb00, 0xc005f2c180) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:810 +0x285 created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).run /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:547 +0x859 goroutine 3963 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch.func1(0x178, 0x0, 0x0, 0x0, 0x0, 0xc00257d500, 0xc00257d440, 0xc003195570) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:353 +0xef created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:342 +0x43d goroutine 4319 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe.func1(0x4d56da0, 0xc0062240c0, 0xc0015ade10, 0xc006228000) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:32 +0x48 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:31 +0x125 goroutine 4293 [select]: github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watchChan).run(0xc001454b60) /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:162 +0x1d6 created by github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3.(*watcher).Watch /go/src/github.com/rancher/k3s/vendor/k8s.io/apiserver/pkg/storage/etcd3/watcher.go:122 +0x145 goroutine 3972 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream.func5(0xc000cdaa80, 0xc004466360, 0x4d56e60, 0xc004468090) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:319 +0xda created by github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:318 +0xbf0 goroutine 3974 [select]: github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc0028185b0, 0xc000a9e4d0, 0xc00463f260) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:810 +0x285 created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).run /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:547 +0x859 goroutine 3975 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc004113090, 0xc001438010, 0x5, 0x5, 0x0, 0x0, 0xc001f17040) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:174 +0xcb github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc004113090, 0xc001438010, 0x5, 0x5, 0xc001f16f80, 0x47c8228, 0xc005c02778) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:168 +0x211 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0044682a0, 0xc001438010, 0x5, 0x5, 0xc000781400, 0xc005c027a0, 0x9742dc) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55 io.ReadAtLeast(0x4c9cbe0, 0xc0044682a0, 0xc001438010, 0x5, 0x5, 0x5, 0x407fb80, 0xc001f88170, 0x3d) /usr/local/go/src/io/io.go:314 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:333 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001efa400, 0xc001438010, 0x5, 0x5, 0x3ffb940, 0x8, 0x7f555fff1f28) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xd2 github.com/rancher/k3s/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001438000, 0x400000, 0x40f230, 0xc001f4a750, 0x90, 0x90, 0x4207f80, 0xc005c02970) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/rancher/k3s/vendor/google.golang.org/grpc.recvAndDecompress(0xc001438000, 0xc001efa400, 0x0, 0x0, 0x400000, 0x0, 0x0, 0x0, 0xc000797af0, 0x10, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/rancher/k3s/vendor/google.golang.org/grpc.recv(0xc001438000, 0x7f556078bf18, 0x70524e8, 0xc001efa400, 0x0, 0x0, 0x437be20, 0xc000797b10, 0x400000, 0x0, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9b github.com/rancher/k3s/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0007e00c0, 0x437be20, 0xc000797b10, 0x0, 0x0) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:1474 +0x156 github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchServer).Recv(0xc000797ac0, 0x4d56da0, 0xc001fb5140, 0xc004192770) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3745 +0x62 github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server.(*KVServerBridge).Watch(0xc000286660, 0x4d85520, 0xc000797ac0, 0x0, 0x0) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server/watch.go:26 +0x142 github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb._Watch_Watch_Handler(0x43577c0, 0xc000286660, 0x4d73dc0, 0xc0007e00c0, 0x70524e8, 0xc001efa400) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3726 +0xad github.com/rancher/k3s/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00124a300, 0x4d8a9e0, 0xc003e77980, 0xc001efa400, 0xc0010daa20, 0x6d62ea0, 0x0, 0x0, 0x0) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/server.go:1237 +0xcdd github.com/rancher/k3s/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00124a300, 0x4d8a9e0, 0xc003e77980, 0xc001efa400, 0x0) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/server.go:1317 +0xccd github.com/rancher/k3s/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc00063eff0, 0xc00124a300, 0x4d8a9e0, 0xc003e77980, 0xc001efa400) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/server.go:722 +0xa5 created by github.com/rancher/k3s/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/server.go:720 +0xa5 goroutine 3976 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server.(*watcher).Start.func1(0xc001fb4f80, 0x2c, 0x4d56da0, 0xc001fb5140, 0xc001f17080, 0x1a, 0xc004192770) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server/watch.go:74 +0x194 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server.(*watcher).Start /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/server/watch.go:63 +0x346 goroutine 3977 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe.func1(0x4d56da0, 0xc001fb52c0, 0xc0015ade10, 0xc00445ad80) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:32 +0x48 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:31 +0x125 goroutine 3978 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch.func1(0xc00445ad20, 0xc00445ad80, 0xc00445ad01, 0xc001f17080, 0x1a) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:343 +0x85 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:341 +0x11d goroutine 4004 [select]: github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc0042d4750) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:530 +0x385 created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:283 +0x305 goroutine 4007 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0037a8eb0, 0xc0008af2b0, 0x5, 0x5, 0x44b1b3, 0xc0044fb318, 0x1) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xd3 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0037a8eb0, 0xc0008af2b0, 0x5, 0x5, 0xc001a3bb00, 0x47c8228, 0xc005c17960) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x190 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0060be630, 0xc0008af2b0, 0x5, 0x5, 0xc000700c00, 0xc005c17988, 0x972e3c) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55 io.ReadAtLeast(0x4c9cbe0, 0xc0060be630, 0xc0008af2b0, 0x5, 0x5, 0x5, 0xc001a3bd80, 0xc0003f9292, 0x0) /usr/local/go/src/io/io.go:314 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:333 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001700900, 0xc0008af2b0, 0x5, 0x5, 0x3ffb940, 0x2, 0x2) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xd2 github.com/rancher/k3s/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0008af2a0, 0x7fffffff, 0xc0003f9290, 0x6, 0x6, 0x0, 0x0, 0x4d22160) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/rancher/k3s/vendor/google.golang.org/grpc.recvAndDecompress(0xc0008af2a0, 0xc001700900, 0x0, 0x0, 0x7fffffff, 0x0, 0x0, 0x0, 0xc0003f9201, 0x0, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/rancher/k3s/vendor/google.golang.org/grpc.recv(0xc0008af2a0, 0x7f556078bf18, 0x70524e8, 0xc001700900, 0x0, 0x0, 0x4408c80, 0xc0037a9040, 0x7fffffff, 0x0, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9b github.com/rancher/k3s/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000b28200, 0x4408c80, 0xc0037a9040, 0x0, 0xc005c17d20, 0x44b1b3) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:885 +0xee github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000b28200, 0xc005c17d60, 0x46a2b7) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:736 +0x46 github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0060c0120, 0xc005c17e10, 0xc005c17de0, 0x44bbff, 0xc0044faf00) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:590 +0x37e github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0060c0120, 0x4408c80, 0xc0037a9040, 0x50, 0x438a820) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:735 +0x105 github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredClientStream).RecvMsg(0xc0008af2c0, 0x4408c80, 0xc0037a9040, 0xc005c17ef4, 0x2) /go/src/github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client_metrics.go:160 +0x4b github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc0007bcf40, 0xc005c17ef0, 0x2, 0x0) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3704 +0x62 github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc0042d4750, 0x4d854c0, 0xc0007bcf40) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:764 +0x5c created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).newWatchClient /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:919 +0x545 goroutine 4006 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream.func5(0xc000cdb880, 0xc0060c0120, 0x4d56e60, 0xc0060be5a0) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:319 +0xda created by github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:318 +0xbf0 goroutine 4008 [select]: github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc0042d4750, 0xc000dd6840, 0xc0044fb0e0) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:810 +0x285 created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).run /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:547 +0x859 goroutine 3957 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe.func1(0x4d56da0, 0xc0039ae680, 0xc0015ade10, 0xc00257d4a0) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:32 +0x48 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster.(*Broadcaster).Subscribe /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/broadcaster/broadcaster.go:31 +0x125 goroutine 3958 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch.func1(0xc00257d440, 0xc00257d4a0, 0xc00257d401, 0xc001e84f60, 0x19) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:343 +0x85 created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog.(*SQLLog).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/sqllog/sql.go:341 +0x11d goroutine 4388 [select]: context.propagateCancel.func1(0x4d58160, 0xc0037bc2c0, 0x4cf5ce0, 0xc004470140) /usr/local/go/src/context/context.go:279 +0xda created by context.propagateCancel /usr/local/go/src/context/context.go:278 +0x1fb goroutine 3981 [chan receive]: github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch.func1(0x178, 0x0, 0x0, 0x0, 0x0, 0xc00445ade0, 0xc00445ad20, 0xc000797c70) /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:353 +0xef created by github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured.(*LogStructured).Watch /go/src/github.com/rancher/k3s/vendor/github.com/k3s-io/kine/pkg/logstructured/logstructured.go:342 +0x43d goroutine 4017 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0037a9540, 0xc001206670, 0x5, 0x5, 0x44b1b3, 0xc0044fba38, 0x1) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:186 +0xd3 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0037a9540, 0xc001206670, 0x5, 0x5, 0xc002866a40, 0x47c8228, 0xc006393960) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:166 +0x190 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0060bf2f0, 0xc001206670, 0x5, 0x5, 0xc000b45800, 0xc006393988, 0x972e3c) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:479 +0x55 io.ReadAtLeast(0x4c9cbe0, 0xc0060bf2f0, 0xc001206670, 0x5, 0x5, 0x5, 0xc001f48700, 0xc001f0a882, 0x0) /usr/local/go/src/io/io.go:314 +0x87 io.ReadFull(...) /usr/local/go/src/io/io.go:333 github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001701300, 0xc001206670, 0x5, 0x5, 0x3ffb940, 0x2, 0x2) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/internal/transport/transport.go:463 +0xd2 github.com/rancher/k3s/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001206660, 0x7fffffff, 0xc001f0a880, 0x6, 0x6, 0x0, 0x0, 0x4d22160) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/rancher/k3s/vendor/google.golang.org/grpc.recvAndDecompress(0xc001206660, 0xc001701300, 0x0, 0x0, 0x7fffffff, 0x0, 0x0, 0x0, 0xc001f0a801, 0x0, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/rancher/k3s/vendor/google.golang.org/grpc.recv(0xc001206660, 0x7f556078bf18, 0x70524e8, 0xc001701300, 0x0, 0x0, 0x4408c80, 0xc003863770, 0x7fffffff, 0x0, ...) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/rpc_util.go:707 +0x9b github.com/rancher/k3s/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000b28600, 0x4408c80, 0xc003863770, 0x0, 0xc006393d20, 0x44b1b3) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:885 +0xee github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000b28600, 0xc006393d60, 0x46a2b7) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:736 +0x46 github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0060c19e0, 0xc006393e10, 0xc006393de0, 0x44bbff, 0xc0044fb620) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:590 +0x37e github.com/rancher/k3s/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0060c19e0, 0x4408c80, 0xc003863770, 0x50, 0x438a820) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:735 +0x105 github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus.(*monitoredClientStream).RecvMsg(0xc0012066a0, 0x4408c80, 0xc003863770, 0xc006393ef4, 0x2) /go/src/github.com/rancher/k3s/vendor/github.com/grpc-ecosystem/go-grpc-prometheus/client_metrics.go:160 +0x4b github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc0007bdb60, 0xc006393ef0, 0x2, 0x0) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/etcdserver/etcdserverpb/rpc.pb.go:3704 +0x62 github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc0042d49c0, 0x4d854c0, 0xc0007bdb60) /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:764 +0x5c created by github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).newWatchClient /go/src/github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3/watch.go:919 +0x545 goroutine 4016 [select]: github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream.func5(0xc001e7ca80, 0xc0060c19e0, 0x4d56e60, 0xc0060bf260) /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:319 +0xda created by github.com/rancher/k3s/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/rancher/k3s/vendor/google.golang.org/grpc/stream.go:318 +0xbf0 goroutine 4034 [select]: github.com/rancher/k3s/vendor/go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc0042d49c0, 0xc000dd6W0902 21:38:14.452598 7 proxier.go:651] Failed to read file /lib/modules/5.13.13_1/modules.builtin with error open /lib/modules/5.13.13_1/modules.builtin: no such file or directory. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules ```

Which OS & Architecture

Which version of k3d

Which version of docker

Server: Engine: Version: 20.10.6 API version: 1.41 (minimum version 1.12) Go version: go1.16.3 Git commit: tag v20.10.6 Built: Mon May 3 18:45:22 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.5.2 GitCommit: UNSET runc: Version: unknown GitCommit:
docker-init: Version: 0.19.0 GitCommit:

Client: Context: default Debug Mode: false

Server: Containers: 2 Running: 1 Paused: 0 Stopped: 1 Images: 2 Server Version: 20.10.6 Storage Driver: btrfs Build Version: Btrfs v5.9 Library Version: 102 Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: UNSET runc version: init version: Security Options: seccomp Profile: default cgroupns Kernel Version: 5.13.13_1 Operating System: void OSType: linux Architecture: x86_64 CPUs: 24 Total Memory: 62.75GiB Name: moon ID: VJHU:2SUH:NJDU:A7OC:YDYH:X3UX:SXVV:I4VQ:WGPC:XW2B:TSKW:NJYA Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false


[logs.txt](https://github.com/rancher/k3d/files/7101982/logs.txt)
iwilltry42 commented 3 years ago

Hi @TeddyDD , thanks for opening this issue! I actually do not suspect btrfs to be causing the problem here, as the Fatal log line says F0902 21:53:19.147300 7 server.go:181] cannot set feature gate SupportPodPidsLimit to false, feature is locked to true (followed by the stacktrace)

Googling this brings up this issue: https://github.com/k3s-io/k3s/issues/3249 As that's not related to armhf (you're also on x86_64), I guess it's an issue with the cgroups setup on your machine. You can try and check that using k3s' check-config functionality as mentioned in the linked issue. :thinking: