Open JasonFengJ9 opened 1 year ago
jdk.internal.platform.Metrics.systemMetrics().getKernelMemoryLimit() is not returning the expected value. @babsingh fyi
The test passes for JDK19 using the below cmd:
make test TEST="jtreg:test/jdk/jdk/internal/platform/docker/TestDockerMemoryMetrics.java" JTREG="JAVA_OPTIONS=--enable-preview -Dvm.continuations=true -Ddocker.support=true;VERBOSE=all"
The reported failure is JDK8 specific. But, the above cmd does not work in the JDK8 extension repo. @llxia Would you know the cmd to run the test in the JDK8 extension repo?
Adapted the following instructions to run the failing test: https://github.com/eclipse-openj9/openj9/wiki/Reproducing-Test-Failures-Locally#run-specific-test-class-from-openjdk-tests-on-jdk11-linux-s390x-openj9-sdk.
With OpenJ9 JDK8, TestDockerMemoryMetrics
fails with the below error. The Docker image (registry.access.redhat.com/ubi8/ubi:latest
) used by the test doesn't have the required GLIBC to support OpenJ9. So, this test should be disabled for OpenJ9 JDK8.
STDERR:
stdout: [JVMJ9VM011W Unable to load j9gc29: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /jdk/jre/lib/amd64/default/libj9gc29.so)
JVMJ9VM011W Unable to load j9vrb29: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /jdk/jre/lib/amd64/default/libj9vrb29.so)
Unhandled exception
Type=Segmentation error vmState=0xffffffff
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=00007F373AC048D0 Handler2=00007F373A2551D0 InaccessibleAddress=0000000000000538
RDI=00007F373400DC40 RSI=000000000000000A RAX=00007F3734037D40 RBX=00007F3734037E18
RCX=0000000000000006 RDX=00007F373C234208 R8=00007F373976DF60 R9=0000000000000005
R10=0000000000000003 R11=00007F373BBB8F80 R12=0000000000000000 R13=00007F373C234208
R14=0000000000000003 R15=0000000000000003
RIP=00007F3738B36845 GS=0000 FS=0000 RSP=00007F373C234200
EFlags=0000000000010206 CS=0033 RBP=00007F373400DC40 ERR=0000000000000004
TRAPNO=000000000000000E OLDMASK=0000000000000000 CR2=0000000000000538
xmm0 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm1 49656e69756e6547 (f: 1970169216.000000, d: 3.823471e+45)
xmm2 0680810101000800 (f: 16779264.000000, d: 2.327561e-277)
xmm3 ffff000000000000 (f: 0.000000, d: -nan)
xmm4 6a2f3a72616a2e73 (f: 1634348672.000000, d: 3.059677e+203)
xmm5 6a2e65636a2f6269 (f: 1781490304.000000, d: 2.978134e+203)
xmm6 80046d8db7654321 (f: 3076866816.000000, d: -6.157814e-309)
xmm7 00007f3738223aa0 (f: 941767296.000000, d: 6.910750e-310)
xmm8 dddddddddddddddd (f: 3722305024.000000, d: -1.456816e+144)
xmm9 ff00ffffffffffff (f: 4294967296.000000, d: -5.829007e+303)
xmm10 2000202020202020 (f: 538976256.000000, d: 1.503368e-154)
xmm11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
xmm15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/jdk/jre/lib/amd64/default/libj9jit29.so
Module_base_address=00007F373892E000
Target=2_90_20230110_000000 (Linux 4.15.0-200-generic)
CPU=amd64 (8 logical CPUs) (0x3ea73a000 RAM)
JVMDUMP039I Processing dump event "gpf", detail "" at 2023/01/11 16:32:46 - please wait.
JVMDUMP032I JVM requested System dump using '//core.20230111.163246.1.0001.dmp' in response to an event
JVMPORT030W /proc/sys/kernel/core_pattern setting "|/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" specifies that the core dump is to be piped to an external program. Attempting to rename either core or core.9. Review the manual for the external program to find where the core dump is written and ensure the program does not truncate it.
JVMDUMP012E Error in System dump: The core file created by child process with pid = 9 was not found. Expected to find core file with name "//core"
JVMDUMP032I JVM requested Java dump using '//javacore.20230111.163246.1.0002.txt' in response to an event
JVMDUMP032I JVM requested Snap dump using '//Snap.20230111.163246.1.0003.trc' in response to an event
JVMDUMP009I Snap dump not available
JVMDUMP013I Processed dump event "gpf", detail "".
];
stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
]
The reported failure in https://github.com/eclipse-openj9/openj9/issues/16460#issue-1494255257 is seen with IBM-J9 (Semeru) JDK8. It is also seen with the RI's JDK8. The API is completely written in Java with no native dependency to the OMR cgroup functions: jdk/internal/platform/cgroupv1/Metrics.java#L449-L452.
This failure is not seen with OpenJ9 JDK19 since Metrics.getKernelMemoryLimit
has been removed. The kernel memory limit metric has been deprecated by the Linux kernel. Related commit: https://github.com/ibmruntimes/openj9-openjdk-jdk19/commit/ce7bf63e67eff780630c114a9a6999f897c1e303.
Backporting the above JDK19 change to https://github.com/ibmruntimes/openj9-openjdk-jdk8 won't help since IBM-J9 (Semeru) uses a different JCL repo.
Also, JDK8 jdk/internal/platform/cgroupv1/Metrics
implementation only supports cgroup-v1. If the above test is run on a cgroup-v2 machine, then it will fail.
The Docker image (registry.access.redhat.com/ubi8/ubi:latest) used by the test doesn't have the required GLIBC to support OpenJ9. So, this test should be disabled for OpenJ9 JDK8.
Proper OpenJ9 builds for jdk8 require glibc 2.12 since it's compiled on Cent 6. Not sure where the OpenJ9 build comes from that needs 2.27. Seems it was built on Ubuntu 18.04, which provides glibc 2.27.
Proper OpenJ9 builds for jdk8 require glibc 2.12 since it's compiled on Cent 6. Not sure where the OpenJ9 build comes from that needs 2.27. Seems it was built on Ubuntu 18.04, which provides glibc 2.27.
Yes, the OpenJ9 JDK8 was built on my local build environment, which is on a Ubuntu 18.04 machine.
It is also seen with the RI's JDK8.
It seems weird that a test provided with OpenJDK 8 doesn't work on OpenJDK 8. I wonder if the Adoptium test runs are running into the same failure? I don't see the test excluded for non-OpenJ9.
If the above test is run on a cgroup-v2 machine, then it will fail.
Is that's what's going on? The test requirement is to run on a cgroup-v1 machine?
Actions to take:
Is that's what's going on? The test requirement is to run on a cgroup-v1 machine?
- There are no test requirements which prevent the test to run on a cgroup-v2 machine: https://github.com/ibmruntimes/openj9-openjdk-jdk8/blob/openj9/jdk/test/jdk/internal/platform/docker/TestDockerMemoryMetrics.java#L30.
- Backporting cgroup-v2 support for the API from JDK19 to JDK8 is complex and challenging. So, we should add test code to prevent running the test on a cgroup v2 machine. cgroup-v2 test is pretty simple: https://github.com/eclipse/omr/blob/69b3a333f847bcc66b325f0f4f65d4e605e7e6e6/port/unix/omrsysinfo.c#L5834-L5844.
@llxia is this test running/passing on Adoptium builds?
I wonder if the Adoptium test runs are running into the same failure? I don't see the test excluded for non-OpenJ9.
It seems that Adoptium does not have the machine for aarch64, but I found the build on linux x86 (failed): https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_dev.openjdk_x86-64_linux/10/consoleFull
00:06:54.192 STDERR:
00:06:54.192 stdout: [[kernelmem, 100m]
00:06:54.192 Exception in thread "main" java.lang.RuntimeException: Kernel Memory limit not equal, expected : [104857600], got : [-1]
00:06:54.192 at MetricsMemoryTester.testKernelMemoryLimit(MetricsMemoryTester.java:114)
00:06:54.192 at MetricsMemoryTester.main(MetricsMemoryTester.java:38)
00:06:54.192 ];
00:06:54.192 stderr: [WARNING: Specifying a kernel memory limit is deprecated and will be removed in a future release.
00:06:54.192 WARNING: You specified a kernel memory limit on a kernel older than 4.0. Kernel memory limits are experimental on older kernels, it won't work as expected and can cause your system to be unstable.
00:06:54.192 ]
00:06:54.192 exitValue = 1
00:06:54.192
00:06:54.192 java.lang.RuntimeException: Expected to get exit value of [0]
00:06:54.192
00:06:54.192 at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:396)
00:06:54.192 at TestDockerMemoryMetrics.testKernelMemoryLimit(TestDockerMemoryMetrics.java:116)
00:06:54.192 at TestDockerMemoryMetrics.main(TestDockerMemoryMetrics.java:58)
00:06:54.192 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
00:06:54.192 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
00:06:54.192 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
00:06:54.192 at java.lang.reflect.Method.invoke(Method.java:498)
00:06:54.192 at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
00:06:54.192 at java.lang.Thread.run(Thread.java:750)
00:06:54.192
00:06:54.192 JavaTest Message: Test threw exception: java.lang.RuntimeException
00:06:54.192 JavaTest Message: shutting down test
00:06:54.192
00:06:54.192
00:06:54.192 TEST RESULT: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: Expected to get exit value of [0]
00:06:54.192 --------------------------------------------------
00:07:27.264 Test results: passed: 5; failed: 1
Seems like the test isn't running at Adoptium since there isn't a machine to run it on. https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_dev.openjdk_aarch64_linux/
is the test requirement is to run on a cgroup-v1 machine?
I'm asking if the docker container we are running the test on is cgroup-v2, and that could explain the failure?
I don't think we should be backporting anything from jdk19. We will provide what OpenJDK 8 provides. If the test/code doesn't work, it can be excluded until OpenJDK fixes it.
I'm asking if the docker container we are running the test on is cgroup-v2, and that could explain the failure?
The test will fail in both Docker containers:
Metrics
instance due to the missing cgroup-v2 implementation. The error output is as follows:
STDERR:
stdout: [[memory, 200m]
Exception in thread "main" java.lang.NullPointerException
at MetricsMemoryTester.testMemoryLimit(MetricsMemoryTester.java:55)
at MetricsMemoryTester.main(MetricsMemoryTester.java:32)
];
stderr: [WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
]
exitValue = 1
java.lang.RuntimeException: Expected to get exit value of [0]
at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:396)
at TestDockerMemoryMetrics.testMemoryLimit(TestDockerMemoryMetrics.java:81)
at TestDockerMemoryMetrics.main(TestDockerMemoryMetrics.java:52)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
at java.lang.Thread.run(Thread.java:825)
JavaTest Message: Test threw exception: java.lang.RuntimeException JavaTest Message: shutting down test
I don't think we should be backporting anything from jdk19. We will provide what OpenJDK 8 provides. If the test/code doesn't work, it can be excluded until OpenJDK fixes it.
Opened https://github.com/adoptium/aqa-tests/pull/4242 to exclude the test until OpenJDK fixes the issue.
Was an upstream (OpenJDK) bug raised for this test case issue? I do not see it referenced anywhere but see the comment
it can be excluded until OpenJDK fixes it
If it exists, we can use that issue number in the excludes PR in https://github.com/adoptium/aqa-tests/pull/4242
https://github.com/adoptium/aqa-tests/pull/4242 has been merged. @pshipton Can this issue be closed, or will it stay open until OpenJDK fixes the issue?
re https://github.com/eclipse-openj9/openj9/issues/16460#issuecomment-1382438843: Responded in https://github.com/adoptium/aqa-tests/pull/4242#issuecomment-1382444134.
Since the test is excluded, this issue (or another) needs to remain open to track the exclude.
[2023-01-23T17:54:04.587Z] STDERR:
[2023-01-23T17:54:04.587Z] stdout: [[kernelmem, 100m]
[2023-01-23T17:54:04.587Z] Exception in thread "main" java.lang.RuntimeException: Kernel Memory limit not equal, expected : [104857600], got : [-1]
[2023-01-23T17:54:04.587Z] at MetricsMemoryTester.testKernelMemoryLimit(MetricsMemoryTester.java:114)
[2023-01-23T17:54:04.587Z] at MetricsMemoryTester.main(MetricsMemoryTester.java:38)
[2023-01-23T17:54:04.587Z] ];
[2023-01-23T17:54:04.587Z] stderr: [WARNING: Specifying a kernel memory limit is deprecated and will be removed in a future release.
[2023-01-23T17:54:04.587Z] ]
[2023-01-23T17:54:04.587Z] exitValue = 1
[2023-01-23T17:54:04.587Z]
[2023-01-23T17:54:04.587Z] java.lang.RuntimeException: Expected to get exit value of [0]
[2023-01-23T17:54:04.587Z]
[2023-01-23T17:54:04.587Z] at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:396)
[2023-01-23T17:54:04.587Z] at TestDockerMemoryMetrics.testKernelMemoryLimit(TestDockerMemoryMetrics.java:116)
[2023-01-23T17:54:04.587Z] at TestDockerMemoryMetrics.main(TestDockerMemoryMetrics.java:58)
[2023-01-23T17:54:04.587Z] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2023-01-23T17:54:04.587Z] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2023-01-23T17:54:04.587Z] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2023-01-23T17:54:04.587Z] at java.lang.reflect.Method.invoke(Method.java:498)
[2023-01-23T17:54:04.587Z] at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
[2023-01-23T17:54:04.587Z] at java.lang.Thread.run(Thread.java:826)
[2023-01-23T17:54:04.587Z]
[2023-01-23T17:54:04.587Z] JavaTest Message: Test threw exception: java.lang.RuntimeException
It seems 0.36
test branch w/o the excludes.
The above build uses aqa-test's v0.9.6-release branch, and this branch does not contain the changes from https://github.com/adoptium/aqa-tests/pull/4242. @llxia @JasonFengJ9 @pshipton Do we need to backport this PR to aqa-test's v0.9.6-release branch?
It would be nice, but not critical.
backport this PR to aqa-test's v0.9.6-release branch?
PR to backport the changes: https://github.com/adoptium/aqa-tests/pull/4275
@pshipton @DanHeidinga The issue resides in OpenJDK8's extension repo, and it is seen in both RI and OpenJ9. Key details are mentioned from https://github.com/eclipse-openj9/openj9/issues/16460#issuecomment-1382309251 and onwards. We have disabled the test and await OpenJDK to fix the issue in JDK8.
It seems maybe we are trying to run the test when not in the correct environment where it can pass. @llxia I've added the test label, and moving this forward.
There are two issues https://github.com/eclipse-openj9/openj9/issues/16460#issuecomment-1382336125:
On cgroup v1 machines, a functional issue will be seen related to Metrics.getKernelMemoryLimit
because the OpenJDK impl in the extension repo uses the deprecated Linux kernel memory limit metric https://github.com/eclipse-openj9/openj9/issues/16460#issuecomment-1382309251. This requires the OpenJDK fix to be backported to JDK8. The fix just removes the API and test code related to the deprecated metrics.
On cgroup v2 machines, the Metrics API won't work in JDK8 because there is only support for cgroup v1. cgroup v2 support was introduced in JDK11. In JDK8, tests related to the Metrics API should only be run on cgroup v1 machines. They will fail on a cgroup v2 machine due to lack of support.
Failure link
From an internal build(
ub18-aarch64-4
):Rerun in Grinder - Change TARGET to run only the failed test targets.
Optional info
Failure output (captured from console output)
10x grinder - all failed