search

seaweedfs / seaweedfs-csi-driver

SeaweedFS CSI Driver https://github.com/seaweedfs/seaweedfs
Apache License 2.0
210 stars 50 forks source link

OOM-Killer invoked when writing data #125

Closed Ruakij closed 1 year ago

Ruakij commented 1 year ago

Description

I have installed the default Helm-Chart. For testing i have spawned 2 debian-slim containers and attached a volume to both of them. When writing to any of them, the weed mount gets killed by oom-killer.

Setup

PersistentVolumeClaim ```yml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: debian namespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: seaweedfs-storage ```
Deployment ```yml # https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ apiVersion: apps/v1 kind: Deployment metadata: name: debian namespace: default labels: app: debian spec: selector: matchLabels: app: debian replicas: 2 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: debian spec: # initContainers: # Init containers are exactly like regular containers, except: # - Init containers always run to completion. # - Each init container must complete successfully before the next one starts. containers: - name: debian image: debian:bookworm-20221219-slim imagePullPolicy: IfNotPresent command: ["sleep", "999999"] resources: requests: cpu: 100m memory: 100Mi limits: cpu: 100m memory: 100Mi volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: debian readOnly: false restartPolicy: Always ```

Steps

Write 1GB random content to /data in 1MB blocks.

root@debian-7bd547b4c-4c8dx:/# dd if=/dev/random of=/data/testfile bs=1M count=1024
dd: error writing 'testfile': Transport endpoint is not connected
dd: closing output file 'testfile': Transport endpoint is not connected

After this, no other container has or can mount /data anymore on this node.

Logs:

journalctl ```log Jun 22 21:33:34 system kernel: weed invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-997 Jun 22 21:33:34 system kernel: CPU: 0 PID: 80597 Comm: weed Not tainted 5.10.0-23-amd64 #1 Debian 5.10.179-1 Jun 22 21:33:34 system kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014 Jun 22 21:33:34 system kernel: Call Trace: Jun 22 21:33:34 system kernel: dump_stack+0x6b/0x83 Jun 22 21:33:34 system kernel: dump_header+0x4a/0x1f4 Jun 22 21:33:34 system kernel: oom_kill_process.cold+0xb/0x10 Jun 22 21:33:34 system kernel: out_of_memory+0x1bd/0x4e0 Jun 22 21:33:34 system kernel: mem_cgroup_out_of_memory+0x138/0x150 Jun 22 21:33:34 system kernel: try_charge+0x762/0x7e0 Jun 22 21:33:34 system kernel: ? __alloc_pages_nodemask+0x161/0x310 Jun 22 21:33:34 system kernel: mem_cgroup_charge+0x7f/0x240 Jun 22 21:33:34 system kernel: handle_mm_fault+0xed3/0x1c00 Jun 22 21:33:34 system kernel: do_user_addr_fault+0x1b8/0x400 Jun 22 21:33:34 system kernel: ? switch_fpu_return+0x44/0xc0 Jun 22 21:33:34 system kernel: exc_page_fault+0x78/0x160 Jun 22 21:33:34 system kernel: ? asm_exc_page_fault+0x8/0x30 Jun 22 21:33:34 system kernel: asm_exc_page_fault+0x1e/0x30 Jun 22 21:33:34 system kernel: RIP: 0033:0x473f1c Jun 22 21:33:34 system kernel: Code: 4c 01 de 48 29 c3 c5 fe 6f 06 c5 fe 6f 4e 20 c5 fe 6f 56 40 c5 fe 6f 5e 60 48 01 c6 c5 fd 7f 07 c5 fd 7f 4f 20 c5 fd 7f 57 40 fd 7f 5f 60 48 01 c7 48 29 c3 77 cf 48 01 c3 48 01 fb c4 c1 7e Jun 22 21:33:34 system kernel: RSP: 002b:000000c000a91b00 EFLAGS: 00010202 Jun 22 21:33:34 system kernel: RAX: 0000000000000080 RBX: 0000000000007fe0 RCX: 000000c001b40200 Jun 22 21:33:34 system kernel: RDX: 0000000000080000 RSI: 000000c001b38220 RDI: 000000c002bd9fa0 Jun 22 21:33:34 system kernel: RBP: 000000c000a91b68 R08: 0000000000200000 R09: 0000000000080000 Jun 22 21:33:34 system kernel: R10: 000000c002bc2000 R11: 0000000000000020 R12: 000000c000a91bf8 Jun 22 21:33:34 system kernel: R13: 0000000000000000 R14: 000000c000723040 R15: 000000c000746000 Jun 22 21:33:34 system kernel: memory: usage 40960kB, limit 40960kB, failcnt 82 Jun 22 21:33:34 system kernel: swap: usage 0kB, limit 0kB, failcnt 0 Jun 22 21:33:34 system kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope: Jun 22 21:33:34 system kernel: anon 40554496 file 0 kernel_stack 294912 percpu 0 sock 0 shmem 0 file_mapped 0 file_dirty 0 file_writeback 0 anon_thp 0 inactive_anon 40558592 active_anon 0 inactive_file 0 active_file 0 unevictable 0 slab_reclaimable 0 slab_unreclaimable 0 slab 0 workingset_refault_anon 0 workingset_refault_file 0 workingset_activate_anon 0 workingset_activate_file 0 workingset_restore_anon 0 workingset_restore_file 0 workingset_nodereclaim 0 pgfault 4719 pgmajfault 0 pgrefill 66 pgscan 101 pgsteal 0 pgactivate 66 pgdeactivate 66 pglazyfree 0 pglazyfreed 0 thp_fault_alloc 0 thp_collapse_alloc 0 Jun 22 21:33:34 system kernel: Tasks state (memory values in pages): Jun 22 21:33:34 system kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name Jun 22 21:33:34 system kernel: [ 79464] 0 79464 182945 5123 143360 0 -997 seaweedfs-csi-d Jun 22 21:33:34 system kernel: [ 80585] 0 80585 198446 20710 315392 0 -997 weed Jun 22 21:33:34 system kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,task=weed,pid=80585,uid=0 Jun 22 21:33:34 system kernel: Memory cgroup out of memory: Killed process 80585 (weed) total-vm:793784kB, anon-rss:32628kB, file-rss:50212kB, shmem-rss:0kB, UID:0 pgtables:308kB oom_score_adj:-997 Jun 22 21:33:34 system systemd[1]: cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope: A process of this unit has been killed by the OOM killer. Jun 22 21:33:34 system systemd[80984]: var-lib-kubelet-plugins-kubernetes.io-csi-seaweedfs\x2dcsi\x2ddriver-d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7-globalmount.mount: Succeeded. Jun 22 21:33:34 system systemd[1]: var-lib-kubelet-plugins-kubernetes.io-csi-seaweedfs\x2dcsi\x2ddriver-d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7-globalmount.mount: Succeeded. ```
seaweedfs-csi-driver-node - csi-seaweedfs-plugin ```log I0622 19:30:18.021030 nodeserver.go:31 node stage volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:18.021490 mounter_seaweedfs.go:46 mounting [filer:8888] /buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount W0622 19:30:18.021539 mounter_seaweedfs.go:145 VolumeContext 'storage.kubernetes.io/csiProvisionerIdentity' ignored I0622 19:30:18.021823 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount -localSocket=/tmp/seaweedfs-mount-1808450857.sock -cacheDir=/var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -collectionQuotaMB=10240 -filer=filer:8888 -concurrentWriters=32 -collection=pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -filer.path=/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -dataCenter=DE-HZ-FSN1-DC00 -cacheCapacityMB=0] mount point owner uid=0 gid=0 mode=drwxrwxrwx current uid=0 gid=0 I0622 19:30:18.714352 leveldb_store.go:47 filer store dir: /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta I0622 19:30:18.714393 file_util.go:23 Folder /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta Permission: -rwxr-xr-x I0622 19:30:18.919796 mount_std.go:259 mounted filer:8888/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:18.919822 mount_std.go:260 This is SeaweedFS version 30GB 3.47 linux amd64 I0622 19:30:18.921373 nodeserver.go:82 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully staged to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:19.028625 nodeserver.go:92 node publish volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount I0622 19:30:19.029976 nodeserver.go:126 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully published to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount E0622 19:33:34.776086 mounter.go:75 weed mount exit, pid: 15, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount, error: signal: killed ```

Comment

memory: usage 40960kB, limit 40960kB looks like its limited to 40MB which isnt enough in this case? I am not sure where this limit comes from, maybe csiAttacher?

Seaweed also doesnt seem to be bothered that the mount isnt correctly attached anymore, no big errors are thrown, no remounts attempted.

I dont know what is failing as i dont see any pods being rescheduled or kubernetes writing events for anything.

Ruakij commented 1 year ago

Big dumb dumb from me.. i have set ressource limits which arent set by default :D