I have installed the default Helm-Chart. For testing i have spawned 2 debian-slim containers and attached a volume to both of them.
When writing to any of them, the weed mount gets killed by oom-killer.
Setup
PersistentVolumeClaim
```yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: debian
namespace: default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi
storageClassName: seaweedfs-storage
```
Deployment
```yml
# https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
apiVersion: apps/v1
kind: Deployment
metadata:
name: debian
namespace: default
labels:
app: debian
spec:
selector:
matchLabels:
app: debian
replicas: 2
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: debian
spec:
# initContainers:
# Init containers are exactly like regular containers, except:
# - Init containers always run to completion.
# - Each init container must complete successfully before the next one starts.
containers:
- name: debian
image: debian:bookworm-20221219-slim
imagePullPolicy: IfNotPresent
command: ["sleep", "999999"]
resources:
requests:
cpu: 100m
memory: 100Mi
limits:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: data
mountPath: /data
volumes:
- name: data
persistentVolumeClaim:
claimName: debian
readOnly: false
restartPolicy: Always
```
Steps
Write 1GB random content to /data in 1MB blocks.
root@debian-7bd547b4c-4c8dx:/# dd if=/dev/random of=/data/testfile bs=1M count=1024
dd: error writing 'testfile': Transport endpoint is not connected
dd: closing output file 'testfile': Transport endpoint is not connected
After this, no other container has or can mount /data anymore on this node.
Logs:
journalctl
```log
Jun 22 21:33:34 system kernel: weed invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-997
Jun 22 21:33:34 system kernel: CPU: 0 PID: 80597 Comm: weed Not tainted 5.10.0-23-amd64 #1 Debian 5.10.179-1
Jun 22 21:33:34 system kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
Jun 22 21:33:34 system kernel: Call Trace:
Jun 22 21:33:34 system kernel: dump_stack+0x6b/0x83
Jun 22 21:33:34 system kernel: dump_header+0x4a/0x1f4
Jun 22 21:33:34 system kernel: oom_kill_process.cold+0xb/0x10
Jun 22 21:33:34 system kernel: out_of_memory+0x1bd/0x4e0
Jun 22 21:33:34 system kernel: mem_cgroup_out_of_memory+0x138/0x150
Jun 22 21:33:34 system kernel: try_charge+0x762/0x7e0
Jun 22 21:33:34 system kernel: ? __alloc_pages_nodemask+0x161/0x310
Jun 22 21:33:34 system kernel: mem_cgroup_charge+0x7f/0x240
Jun 22 21:33:34 system kernel: handle_mm_fault+0xed3/0x1c00
Jun 22 21:33:34 system kernel: do_user_addr_fault+0x1b8/0x400
Jun 22 21:33:34 system kernel: ? switch_fpu_return+0x44/0xc0
Jun 22 21:33:34 system kernel: exc_page_fault+0x78/0x160
Jun 22 21:33:34 system kernel: ? asm_exc_page_fault+0x8/0x30
Jun 22 21:33:34 system kernel: asm_exc_page_fault+0x1e/0x30
Jun 22 21:33:34 system kernel: RIP: 0033:0x473f1c
Jun 22 21:33:34 system kernel: Code: 4c 01 de 48 29 c3 c5 fe 6f 06 c5 fe 6f 4e 20 c5 fe 6f 56 40 c5 fe 6f 5e 60 48 01 c6 c5 fd 7f 07 c5 fd 7f 4f 20 c5 fd 7f 57 40 fd 7f 5f 60 48 01 c7 48 29 c3 77 cf 48 01 c3 48 01 fb c4 c1 7e
Jun 22 21:33:34 system kernel: RSP: 002b:000000c000a91b00 EFLAGS: 00010202
Jun 22 21:33:34 system kernel: RAX: 0000000000000080 RBX: 0000000000007fe0 RCX: 000000c001b40200
Jun 22 21:33:34 system kernel: RDX: 0000000000080000 RSI: 000000c001b38220 RDI: 000000c002bd9fa0
Jun 22 21:33:34 system kernel: RBP: 000000c000a91b68 R08: 0000000000200000 R09: 0000000000080000
Jun 22 21:33:34 system kernel: R10: 000000c002bc2000 R11: 0000000000000020 R12: 000000c000a91bf8
Jun 22 21:33:34 system kernel: R13: 0000000000000000 R14: 000000c000723040 R15: 000000c000746000
Jun 22 21:33:34 system kernel: memory: usage 40960kB, limit 40960kB, failcnt 82
Jun 22 21:33:34 system kernel: swap: usage 0kB, limit 0kB, failcnt 0
Jun 22 21:33:34 system kernel: Memory cgroup stats for /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope:
Jun 22 21:33:34 system kernel: anon 40554496
file 0
kernel_stack 294912
percpu 0
sock 0
shmem 0
file_mapped 0
file_dirty 0
file_writeback 0
anon_thp 0
inactive_anon 40558592
active_anon 0
inactive_file 0
active_file 0
unevictable 0
slab_reclaimable 0
slab_unreclaimable 0
slab 0
workingset_refault_anon 0
workingset_refault_file 0
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgfault 4719
pgmajfault 0
pgrefill 66
pgscan 101
pgsteal 0
pgactivate 66
pgdeactivate 66
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 0
thp_collapse_alloc 0
Jun 22 21:33:34 system kernel: Tasks state (memory values in pages):
Jun 22 21:33:34 system kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jun 22 21:33:34 system kernel: [ 79464] 0 79464 182945 5123 143360 0 -997 seaweedfs-csi-d
Jun 22 21:33:34 system kernel: [ 80585] 0 80585 198446 20710 315392 0 -997 weed
Jun 22 21:33:34 system kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f448f0_d51b_490b_9754_578fa1f8d0f9.slice/cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope,task=weed,pid=80585,uid=0
Jun 22 21:33:34 system kernel: Memory cgroup out of memory: Killed process 80585 (weed) total-vm:793784kB, anon-rss:32628kB, file-rss:50212kB, shmem-rss:0kB, UID:0 pgtables:308kB oom_score_adj:-997
Jun 22 21:33:34 system systemd[1]: cri-containerd-f5b1f3a01cf6d5cc2acf7073f51da8d2a47922c44b600b5e694b5ced9804ec77.scope: A process of this unit has been killed by the OOM killer.
Jun 22 21:33:34 system systemd[80984]: var-lib-kubelet-plugins-kubernetes.io-csi-seaweedfs\x2dcsi\x2ddriver-d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7-globalmount.mount: Succeeded.
Jun 22 21:33:34 system systemd[1]: var-lib-kubelet-plugins-kubernetes.io-csi-seaweedfs\x2dcsi\x2ddriver-d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7-globalmount.mount: Succeeded.
```
seaweedfs-csi-driver-node - csi-seaweedfs-plugin
```log
I0622 19:30:18.021030 nodeserver.go:31 node stage volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount
I0622 19:30:18.021490 mounter_seaweedfs.go:46 mounting [filer:8888] /buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount
W0622 19:30:18.021539 mounter_seaweedfs.go:145 VolumeContext 'storage.kubernetes.io/csiProvisionerIdentity' ignored
I0622 19:30:18.021823 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount -localSocket=/tmp/seaweedfs-mount-1808450857.sock -cacheDir=/var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -collectionQuotaMB=10240 -filer=filer:8888 -concurrentWriters=32 -collection=pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -filer.path=/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -dataCenter=DE-HZ-FSN1-DC00 -cacheCapacityMB=0]
mount point owner uid=0 gid=0 mode=drwxrwxrwx
current uid=0 gid=0
I0622 19:30:18.714352 leveldb_store.go:47 filer store dir: /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta
I0622 19:30:18.714393 file_util.go:23 Folder /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta Permission: -rwxr-xr-x
I0622 19:30:18.919796 mount_std.go:259 mounted filer:8888/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount
I0622 19:30:18.919822 mount_std.go:260 This is SeaweedFS version 30GB 3.47 linux amd64
I0622 19:30:18.921373 nodeserver.go:82 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully staged to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount
I0622 19:30:19.028625 nodeserver.go:92 node publish volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount
I0622 19:30:19.029976 nodeserver.go:126 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully published to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount
E0622 19:33:34.776086 mounter.go:75 weed mount exit, pid: 15, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount, error: signal: killed
```
Comment
memory: usage 40960kB, limit 40960kB looks like its limited to 40MB which isnt enough in this case?
I am not sure where this limit comes from, maybe csiAttacher?
Seaweed also doesnt seem to be bothered that the mount isnt correctly attached anymore, no big errors are thrown, no remounts attempted.
I dont know what is failing as i dont see any pods being rescheduled or kubernetes writing events for anything.
Description
I have installed the default Helm-Chart. For testing i have spawned 2 debian-slim containers and attached a volume to both of them. When writing to any of them, the
weed mount
gets killed by oom-killer.Setup
PersistentVolumeClaim
```yml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: debian namespace: default spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi storageClassName: seaweedfs-storage ```Deployment
```yml # https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ apiVersion: apps/v1 kind: Deployment metadata: name: debian namespace: default labels: app: debian spec: selector: matchLabels: app: debian replicas: 2 strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: debian spec: # initContainers: # Init containers are exactly like regular containers, except: # - Init containers always run to completion. # - Each init container must complete successfully before the next one starts. containers: - name: debian image: debian:bookworm-20221219-slim imagePullPolicy: IfNotPresent command: ["sleep", "999999"] resources: requests: cpu: 100m memory: 100Mi limits: cpu: 100m memory: 100Mi volumeMounts: - name: data mountPath: /data volumes: - name: data persistentVolumeClaim: claimName: debian readOnly: false restartPolicy: Always ```Steps
Write 1GB random content to /data in 1MB blocks.
After this, no other container has or can mount /data anymore on this node.
Logs:
journalctl
```log Jun 22 21:33:34 system kernel: weed invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=-997 Jun 22 21:33:34 system kernel: CPU: 0 PID: 80597 Comm: weed Not tainted 5.10.0-23-amd64 #1 Debian 5.10.179-1 Jun 22 21:33:34 system kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014 Jun 22 21:33:34 system kernel: Call Trace: Jun 22 21:33:34 system kernel: dump_stack+0x6b/0x83 Jun 22 21:33:34 system kernel: dump_header+0x4a/0x1f4 Jun 22 21:33:34 system kernel: oom_kill_process.cold+0xb/0x10 Jun 22 21:33:34 system kernel: out_of_memory+0x1bd/0x4e0 Jun 22 21:33:34 system kernel: mem_cgroup_out_of_memory+0x138/0x150 Jun 22 21:33:34 system kernel: try_charge+0x762/0x7e0 Jun 22 21:33:34 system kernel: ? __alloc_pages_nodemask+0x161/0x310 Jun 22 21:33:34 system kernel: mem_cgroup_charge+0x7f/0x240 Jun 22 21:33:34 system kernel: handle_mm_fault+0xed3/0x1c00 Jun 22 21:33:34 system kernel: do_user_addr_fault+0x1b8/0x400 Jun 22 21:33:34 system kernel: ? switch_fpu_return+0x44/0xc0 Jun 22 21:33:34 system kernel: exc_page_fault+0x78/0x160 Jun 22 21:33:34 system kernel: ? asm_exc_page_fault+0x8/0x30 Jun 22 21:33:34 system kernel: asm_exc_page_fault+0x1e/0x30 Jun 22 21:33:34 system kernel: RIP: 0033:0x473f1c Jun 22 21:33:34 system kernel: Code: 4c 01 de 48 29 c3 c5 fe 6f 06 c5 fe 6f 4e 20 c5 fe 6f 56 40 c5 fe 6f 5e 60 48 01 c6 c5 fd 7f 07 c5 fd 7f 4f 20 c5 fd 7f 57 40seaweedfs-csi-driver-node - csi-seaweedfs-plugin
```log I0622 19:30:18.021030 nodeserver.go:31 node stage volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:18.021490 mounter_seaweedfs.go:46 mounting [filer:8888] /buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount W0622 19:30:18.021539 mounter_seaweedfs.go:145 VolumeContext 'storage.kubernetes.io/csiProvisionerIdentity' ignored I0622 19:30:18.021823 mounter.go:53 Mounting fuse with command: weed and args: [-logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount -localSocket=/tmp/seaweedfs-mount-1808450857.sock -cacheDir=/var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -collectionQuotaMB=10240 -filer=filer:8888 -concurrentWriters=32 -collection=pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -filer.path=/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d -dataCenter=DE-HZ-FSN1-DC00 -cacheCapacityMB=0] mount point owner uid=0 gid=0 mode=drwxrwxrwx current uid=0 gid=0 I0622 19:30:18.714352 leveldb_store.go:47 filer store dir: /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta I0622 19:30:18.714393 file_util.go:23 Folder /var/cache/seaweedfs/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/2c0f9148/meta Permission: -rwxr-xr-x I0622 19:30:18.919796 mount_std.go:259 mounted filer:8888/buckets/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:18.919822 mount_std.go:260 This is SeaweedFS version 30GB 3.47 linux amd64 I0622 19:30:18.921373 nodeserver.go:82 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully staged to /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount I0622 19:30:19.028625 nodeserver.go:92 node publish volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount I0622 19:30:19.029976 nodeserver.go:126 volume pvc-20afb78d-ce01-40eb-bfef-00544f353c4d successfully published to /var/lib/kubelet/pods/aa6d43e3-02b8-4f34-ae17-389b00b56209/volumes/kubernetes.io~csi/pvc-20afb78d-ce01-40eb-bfef-00544f353c4d/mount E0622 19:33:34.776086 mounter.go:75 weed mount exit, pid: 15, path: /var/lib/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/d19d5bd170bbb36c9edbb90b99bef828e0a6ea08a80ef4290cd68f128f1accd7/globalmount, error: signal: killed ```Comment
memory: usage 40960kB, limit 40960kB
looks like its limited to 40MB which isnt enough in this case? I am not sure where this limit comes from, maybecsiAttacher
?Seaweed also doesnt seem to be bothered that the mount isnt correctly attached anymore, no big errors are thrown, no remounts attempted.
I dont know what is failing as i dont see any pods being rescheduled or kubernetes writing events for anything.