Closed pshipton closed 2 years ago
@tajila fyi
@babsingh Can you take a look at this one
@babsingh I have got an internal sev1 case with the same issue.can you please have a look at it?
Yes, I will take a look.
As per https://github.com/eclipse/omr/issues/1281, we only added support for cgroup v1 in OMR: https://github.com/eclipse/omr/pull/1310. Support for cgroup v2 is still pending. @ashu-mehra Can you point us to any resources (design docs, unmerged code, etc.) which can be used to support cgroup v2 in OMR?
@babsingh I didn't get any chance to explore cgroup v2 mechanism. I did create an issue for it but due to time constraint it never got worked on. The best resource I can refer to would be the man page for cgroups which has description for both versions and take it from there.
@babsingh can you make a clear statement about the workaround?
In JDK17+ extensions repo, there is a Java API which supports Cgroup V1 and V2: https://github.com/ibmruntimes/openj9-openjdk-jdk17/tree/openj9/src/java.base/linux/classes/jdk/internal/platform. There is a possibility to backport this API to Java 8 and Java 11 if this API can replace uses of OMR Cgroup native API within OpenJ9.
This Java API has only one native dependency: CgroupMetrics.isUseContainerSupport -> JVM_IsUseContainerSupport
.
OpenJ9 should be able to use this API since it has its own implementation of JVM_IsUseContainerSupport
.
OpenJ9 uses the OMR Cgroup native API in the following locations:
1. runtime/rasdump/javadump.cpp (writeCgroupMetrics)
OMR Cgroup functions used:
- omrsysinfo_cgroup_get_enabled_subsystems
- omrsysinfo_cgroup_is_system_available
- omrsysinfo_get_cgroup_subsystem_list
- omrsysinfo_cgroup_subsystem_iterator_init
- omrsysinfo_cgroup_subsystem_iterator_hasNext
- omrsysinfo_cgroup_subsystem_iterator_metricKey
- omrsysinfo_cgroup_subsystem_iterator_destroy
OMR Cgroup data structs used:
- OMRCgroupEntry
- OMRCgroupMetricIteratorState
- OMRCgroupMetricElement
2. runtime/gc_base/GCExtensions.cpp
OMR Cgroup functions used:
- omrsysinfo_cgroup_are_subsystems_enabled
- omrsysinfo_cgroup_is_memlimit_set
3. runtime/vm/jvminit.c
OMR Cgroup functions used:
- omrsysinfo_cgroup_enable_subsystems
- omrsysinfo_cgroup_get_available_subsystems
4. runtime/compiler/control/CompilationThread.cpp
OMR Cgroup functions used:
- omrsysinfo_cgroup_are_subsystems_enabled (OMR_CGROUP_SUBSYSTEM_MEMORY)
Invoking the extensions repo Cgroup Java API in the above OpenJ9 locations may not be possible because the above locations won't allow execution of Java code through call-ins. Also, it may not be feasible to translate from the OMR Cgroup native API to the extensions repo Cgroup Java API (missing functionality). So, our best bet is to update the OMR Cgroup native API to support Cgroup v2.
Not sure if the previous is actually a question or not, it's arrived at the right conclusion. The usage in jvminit.c occurs during option processing before java code is running. Similarly for GC and probably for JIT as well.
From @mpirvu:
I discovered that published Semeru container images do not load the embedded AOT code because of various AOT incompatibility issues. This happens because the AOT code is not in "portable" format. This is most likely due to the fact that Semeru images were built in an environment with cgroups v2 and OpenJ9 is not able to detect that is running in containers. I used a machine with cgroups v2 and generated a javacore while running in containers. The javacore showed:
1CICONTINFO Running in container : FALSE
1CICGRPINFO JVM support for cgroups enabled : FALSE
I am raising the priority of this issue since many customers running OpenJ9 in containers will not be able to use AOT and experience a start-up slowdown. As I understand, we do not have much control on the dockerhub process/environment that builds the images.
There is a temporary workaround for this issue: https://github.com/eclipse-openj9/openj9/issues/14190#issuecomment-1010122701. Docker containers can be switched back to cgroup v1 since both cgroup v1 and v2 should be available in newer containers.
The problem is that we don't control the process of building the Semeru images. Once those images have been produced, they contain non-portable AOT.
@mpirvu This work is being given high priority. We should have minimal support by April. This should address customer issues.
For the linked docker issue, is the Max. Heap Size (Estimated): 512.00M
being calculated from j9gc_get_maximum_heap_size
? Where memoryMax
is calculated at https://github.com/eclipse/omr/blob/644b9078b90a441a73da0ab9da66dc60d283033a/gc/base/GCExtensionsBase.cpp#L305-L324
and is assigned a maximum of 512M, then in MM_GCExtensions::computeDefaultMaxHeapForJava
gets expanded if cgroup is enabled: https://github.com/eclipse-openj9/openj9/blob/cc586f03bd5359157f99fb342015f24f4e064755/runtime/gc_base/GCExtensions.cpp#L254-L255
Is this correct?
As you pointed correctly in MM_GCExtensions::computeDefaultMaxHeapForJava()
we set memoryMax
for cgroup and adjust it later to 25% of available RAM if it is larger (for Java 11 and up only, Java 8 has hardcoded 512m limit). I guess you should not touch this logic but make omrsysinfo_cgroup_are_subsystems_enabled()
and omrsysinfo_cgroup_is_memlimit_set()
work for cgroup v2
In omrsysinfo_cgroup_is_memlimit_set
: https://github.com/eclipse/omr/blob/644b9078b90a441a73da0ab9da66dc60d283033a/port/unix/omrsysinfo.c#L5967-L5974
My cgroups v1 machine had a memory.limit_in_bytes
as 9223372036854771712 for a simple java test process which would trigger this error. So basically if the cgroup memlimit is not set we don't expand the heap size (the usablePhysicalMemory
was not obtained from the cgroup memory limit file).
The following was a docker with v1 cgroup:
root@658e20312005:~/hostdir/openj9-openjdk-jdk8# build/linux-x86_64-normal-server-release/images/j2sdk-image/bin/java -XshowSettings:vm -XX:+OriginalJDK8HeapSizeCompatibilityMode -version
VM settings:
Max. Heap Size (Estimated): 512.00M
...
To confirm, the memory limit was not set for this process (125):
root@658e20312005:/# cat /sys/fs/cgroup/memory/cgroup.procs
1
115
125
153
root@658e20312005:/# cat /sys/fs/cgroup/memory/memory.limit_in_bytes
9223372036854771712
So even with cgroup v1 the max heap size is 512M. I'm not sure this is because cgroups v2 is not implemented.
So even with cgroup v1 the max heap size is 512M. I'm not sure this is because cgroups v2 is not implemented.
@EricYangIBM Study the behaviour reported in https://github.com/ibmruntimes/ci.docker/issues/124#issue-1084626215.
@dmitripivkine In https://github.com/ibmruntimes/ci.docker/issues/124#issue-1084626215, Max. Heap Size (Estimated): 3.00G
is shown for J9 Java8. The hardcoded 512m limit is not being enforced for Java8.
Maybe that container has a cgroup limit enforced? That would explain the heap extended by https://github.com/eclipse-openj9/openj9/blob/cc586f03bd5359157f99fb342015f24f4e064755/runtime/gc_base/GCExtensions.cpp#L254-L255
Maybe that container has a cgroup limit enforced?
Yes, I think it is enforced via docker run -m 4GB
.
Task 6 for cgroup v2 is complete (reference: https://github.com/eclipse/omr/issues/1281#issuecomment-1072796875). This means that cgroup v2 has the same functionality as cgroup v1. We should be able to verify if customer issues are fixed.
re https://github.com/eclipse-openj9/openj9/issues/14190#issuecomment-1068441339: @mpirvu Can you please confirm if the not be able to use AOT and experience a start-up slowdown
issue is resolved?
Can you please confirm if the not be able to use AOT and experience a start-up slowdown issue is resolved?
I will once nightly builds are available with the PR that was merged today
I will once nightly builds are available with the PR that was merged today
Referring to https://github.com/eclipse/omr/issues/1281#issuecomment-1072796875, the AOT issue should have been resolved by Tasks 1 and 2. Today's PR only updates the cgroup stats in the javacore.
For the original issue, @EricYangIBM had verified that https://github.com/ibmruntimes/ci.docker/issues/124 was resolved in https://github.com/eclipse/omr/pull/6432#issuecomment-1096810188. So, it should also have been resolved by Tasks 1 and 2 in https://github.com/eclipse/omr/issues/1281#issuecomment-1072796875.
fyi @lgrateau
I verified on a Ubuntu 22.04 machine that the latest nightly build of OpenJ9 correctly determines that is running in container and sees the correct CPU and memory limits (on the same machine, the existing OpenJ9 0.32.0 does not read these limits correctly)
@tajila Why did we move this issue from the 0.33 to 0.34 release? As of now, cgroup v2 support is identical to cgroup v1 support. So, this issue can be closed once the end users verify the fix.
Keeping it to track the tests changes. If you have another item for that we can close this.
If you have another item for that we can close this.
Those tests are OMR specific. In addition to those tests, all pending cgroup related OMR work is being tracked in https://github.com/eclipse/omr/issues/1281#issuecomment-1072796875.
This OpenJ9 issue was primarily opened to address cgroup related user issues. Since those issues have been resolved, we should close this issue.
See https://github.com/ibmruntimes/ci.docker/issues/124