Dynamically provision Stateful Persistent Replicated Cluster-wide Fabric Volumes & Filesystems for Kubernetes that is provisioned from an optimized NVME SPDK backend data storage stack.
Apache License 2.0
750
stars
109
forks
source link
Mixed mount of HugePages leads to process hang #1741
Describe the bug
On k8s, io-engine container hangs whilst trying to flock /dev/hugepages
To Reproduce
Setup your system with default 1GiB hugepage size and allocate 2MiB hugepages.
Expected behavior
Should work :)
Additional context
The problem is that we end up with both 1Gi and 2Mi hugepages mounted on /dev/hugepages.
/ # mount | grep huge
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=1024M)
nodev on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
EAL init is not able to cope with this and hangs as soon as it tries to flock /dev/hugepages.
I think the fix is to ensure we mount 1Gi and 2Mi on separate mounts, example:
- name: hugepage
mountPath: /dev/hugepages-2MiB
In fact today helm-chart is not very flexible as it simply calls out 2MiB hugepages. We should add some vars to allow choosing which hugepages to use...
Describe the bug On k8s, io-engine container hangs whilst trying to flock /dev/hugepages
To Reproduce Setup your system with default 1GiB hugepage size and allocate 2MiB hugepages.
Expected behavior Should work :)
Additional context The problem is that we end up with both 1Gi and 2Mi hugepages mounted on /dev/hugepages.
EAL init is not able to cope with this and hangs as soon as it tries to flock /dev/hugepages.
I think the fix is to ensure we mount 1Gi and 2Mi on separate mounts, example:
In fact today helm-chart is not very flexible as it simply calls out 2MiB hugepages. We should add some vars to allow choosing which hugepages to use...