LuQQiu
commented
1 year ago @Sharathmk99 Thanks for taking the jstat, do you have the OOM error message?
we are trying to figure out whether this is direct memory OOM or heap memory OOM
which can be indicated by OOM error message e.g. OutOfMemoryError: Direct buffer memory
or by adding the following java options -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics to JVM and run jcmd <pid> VM.native_memory. for more details, please refer to https://www.mastertheboss.com/java/troubleshooting-outofmemoryerror-direct-buffer-memory/#:~:text=An%20OutOfMemoryError%3A%20Direct%20buffer%20memory,libraries%20or%20I%2FO%20operations.
Sharathmk99
ssz1997
dbw9580
humengyu2012
github-actions[bot]
Alluxio Version: What version of Alluxio are you using? 2.9.3
Describe the bug A clear and concise description of what the bug is. I installed Alluxio in our Kubernetes cluster using helm chart with one master and one worker configuration. But I keep getting OOM for worker, I increased to 48GB, but still I get OOM after sometime. My worker configuration,
My Properties
My Tierstore
Worker jvmOptions
This is my ConfigMap
Container status,
jstat output when RAM usage was
70GTo Reproduce Steps to reproduce the behavior (as minimally and precisely as possible) Deploy Alluxio cluster with one master and one worker. Configured UFS to use NFS mount of 7TB volume size. Create sample training pod to read data from
fuse.Expected behavior A clear and concise description of what you expected to happen. Should be able to control the Worker RAM memory usage
Urgency Describe the impact and urgency of the bug. Worker keeps restarting every few hours
Are you planning to fix it Please indicate if you are already working on a PR.
Additional context Add any other context about the problem here.