search

adoptium / infrastructure

This repo contains all information about machine maintenance.
Apache License 2.0
87 stars 102 forks source link

JDK8 arm32 TestJlmRemote and jdk_util_2 tests pass only on build-equinix-ubuntu1604-armv7l-1 #2376

Open Haroon-Khel opened 3 years ago

Haroon-Khel commented 3 years ago

TestJlmRemote from sanity system and jdk_util_2 from sanity openjdk seem to pass only on build-equinix-ubuntu1604-armv7l-1 which is our only arm32 machine which isn't a docker container.

Latest TestJlmRemote pass https://ci.adoptopenjdk.net/job/Grinder/2434/console

Error log for latest failure can be found here https://github.com/adoptium/aqa-tests/issues/3117

Latest jdk_util_2 pass https://ci.adoptopenjdk.net/job/Grinder/2429/console

Error log for latest failure can be found here https://github.com/adoptium/aqa-tests/issues/3118

Haroon-Khel commented 3 years ago

Correction, build-equinix-ubuntu1604-armv7l-1 is a docker container. It is hosted on docker-packet-ubuntu2004-armv8-1, an aarch64 machine. But the docker container is arm32

root@eb21061dfa35:/# uname -m
armv7l
Haroon-Khel commented 3 years ago

The 3 arm32 machines, test-docker-ubuntu2004-armv7l-1 -2 and -3, hosted on dockerhost-equinix-ubuntu2004-armv8-1 an aarch64 machine, return aarch64 as their architecture and not armv7l as expected

root@412c7c344fe5:/# uname -m
aarch64
root@db94894974b0:/# uname -m
aarch64
root@1bfd2a038c8d:/# uname -m
aarch64
sxa commented 3 years ago

Correct. They are running a 64-bit kernel so aarch64 is technically correct. They're just not running 32-bit programs.

On the build container I "patched" the uname command to retrun the correct value as I could easily influence the build process - the original uname was renamed to uname.real and then a replacement shell script put in the place of the original uname:

root@test-equinix-ubuntu2004-armv8l-02:~# docker exec build-equinix-ubuntu1604-armv7l-2X cat /bin/uname
/bin/uname.real $@ | sed 's/aarch64/armv7l/g'
root@test-equinix-ubuntu2004-armv8l-02:~#
Haroon-Khel commented 3 years ago

The TestJlmRemote tests may be intermittent as they have just passed on test-docker-ubuntu2004-armv7l-2, which they had previously failed on, https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2473/console

I also ran them on a new ubuntu 2004 docker container on test-docker-ubuntu2004-armv7l-2's host machine. The tests passed.

Rerunning 10 iterations on test-docker-ubuntu2004-armv7l-1 -2 and -3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2475/console https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2476/console https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2477/console

sxa commented 3 years ago

Seems inconsistent on all three systems - TestJlmRemoteMemoryAuth_2 and TestJlmRemoteMemoryNoAuth_2 both seem to fail quite often

sxa commented 3 years ago

Hmmm:

20:15:40  CL2 stderr j> 2021/11/09 20:15:35.906   Peak usage smaller than current usage here:
20:15:40  CL2 stderr j> 2021/11/09 20:15:35.907   Memory Pool:              CodeCache
20:15:40  CL2 stderr j> 2021/11/09 20:15:35.908   Memory Type:              NON_HEAP
20:15:40  CL2 stderr j> 2021/11/09 20:15:35.908   Peak Usage:               init = 1572864(1536K) used = 5734016(5599K) committed = 5832704(5696K) max = 33554432(32768K)
20:15:40  CL2 stderr j> 2021/11/09 20:15:35.908   Current Usage:            init = 1572864(1536K) used = 5775744(5640K) committed = 5832704(5696K) max = 33554432(32768K)
20:15:40  CL2 stderr Exception in thread "main" java.lang.AssertionError: Peak Usage used memory smaller than Current Usage used memory
20:15:40  CL2 stderr    at org.junit.Assert.fail(Assert.java:88)
20:15:40  CL2 stderr    at net.adoptopenjdk.test.jlm.resources.MemoryData.checkPeakAndCurrentMemoryUsage(MemoryData.java:521)
20:15:40  CL2 stderr    at net.adoptopenjdk.test.jlm.resources.MemoryData.writeData(MemoryData.java:401)
20:15:40  CL2 stderr    at net.adoptopenjdk.test.jlm.remote.MemoryProfiler.getStatsViaServer(MemoryProfiler.java:254)
20:15:40  CL2 stderr    at net.adoptopenjdk.test.jlm.remote.MemoryProfiler.main(MemoryProfiler.java:112)
20:15:40  STF 20:15:35.967 - **FAILED** Process CL2 ended with exit code (1) and not the expected exit code/s (0)
20:15:40  STF 20:15:35.967 - Monitoring Report Summary:
20:15:40  STF 20:15:35.967 -   o Process CL2 ended with exit code (1) and not the expected exit code/s (0)
20:15:40  STF 20:15:35.967 -   o Process LT2 is still running as expected

I think Simon looked into something similar - I suggest looking for other issues which might mention Peak Usage used memory smaller than Current Usage used memory

Haroon-Khel commented 3 years ago

Related issue https://github.com/adoptium/aqa-systemtest/issues/274

sxa commented 3 years ago

Note: The failures do also occur on non-docker systems:

13:51:46  CL1 j> 2021/11/10 13:51:45.741 Starting to write data
13:53:57  CL1 stderr j> 2021/11/10 13:53:42.917   Peak usage smaller than current usage here:
13:53:57  CL1 stderr j> 2021/11/10 13:53:42.919   Memory Pool:              CodeCache
13:53:57  CL1 stderr j> 2021/11/10 13:53:42.924   Memory Type:              Non-heap memory
13:53:57  CL1 stderr j> 2021/11/10 13:53:42.927   Peak Usage:               init = 1572864(1536K) used = 4347520(4245K) committed = 4587520(4480K) max = 33554432(32768K)
13:53:57  CL1 stderr j> 2021/11/10 13:53:42.930   Current Usage:            init = 1572864(1536K) used = 4446720(4342K) committed = 4587520(4480K) max = 33554432(32768K)
13:53:57  CL1 stderr Exception in thread "main" java.lang.AssertionError: Peak Usage used memory smaller than Current Usage used memory
13:53:57  CL1 stderr    at org.junit.Assert.fail(Assert.java:88)
13:53:57  CL1 stderr    at net.adoptopenjdk.test.jlm.resources.MemoryData.checkPeakAndCurrentMemoryUsage(MemoryData.java:521)
13:53:57  CL1 stderr    at net.adoptopenjdk.test.jlm.resources.MemoryData.writeData(MemoryData.java:182)
13:53:57  CL1 stderr    at net.adoptopenjdk.test.jlm.remote.MemoryProfiler.getStatsViaProxy(MemoryProfiler.java:183)
13:53:57  CL1 stderr    at net.adoptopenjdk.test.jlm.remote.MemoryProfiler.main(MemoryProfiler.java:110)
13:53:57  STF 13:53:43.409 - **FAILED** Process CL1 ended with exit code (1) and not the expected exit code/s (0)
13:53:57  STF 13:53:43.409 - Monitoring Report Summary:
13:53:57  STF 13:53:43.410 -   o Process CL1 ended with exit code (1) and not the expected exit code/s (0)
Haroon-Khel commented 3 years ago

Rerunning jdk_util_2, 5 iterations to determine whether it is machine specific and or intermittent

https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2656/console test-docker-ubuntu2004-armv7l-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2657/console test-docker-ubuntu2004-armv7l-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2658/console test-docker-ubuntu2004-armv7l-3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2659/console build-equinix-ubuntu1604-armv7l-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/2660/console test-sxamb-armv7l-raspbian-pi4

sxa commented 4 weeks ago

Needs retesting since this was from three years ago

Haroon-Khel commented 2 weeks ago

Rerunning on arm32 nodes

test-docker-ubuntu2004-armv7l-5 https://ci.adoptium.net/job/Grinder/11455/console

Failed tests

TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/RangeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/SplittableRandomTest.java
Test results: passed: 659; failed: 7

test-docker-ubuntu2004-armv7l-4 https://ci.adoptium.net/job/Grinder/11456/console

TEST: java/util/stream/boottest/java/util/stream/FlagOpTest.java
TEST: java/util/stream/boottest/java/util/stream/LongNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/RangeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
Test results: passed: 656; failed: 10

test-docker-ubuntu2004-armv7l-6 https://ci.adoptium.net/job/Grinder/11457/console

TEST: java/util/stream/boottest/java/util/stream/DoubleNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/SplittableRandomTest.java
Test results: passed: 658; failed: 8

test-docker-ubuntu2004-armv7l-1 https://ci.adoptium.net/job/Grinder/11458/console

TEST: java/util/stream/boottest/java/util/stream/DoubleNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/FlagOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/RangeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/boottest/java/util/stream/IntNodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java
Test results: passed: 654; failed: 11; error: 1

test-docker-ubuntu2004-armv7l-2 https://ci.adoptium.net/job/Grinder/11459/console

TEST: java/util/concurrent/ConcurrentHashMap/ConcurrentAssociateTest.java
TEST: java/util/stream/boottest/java/util/stream/DoubleNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/FlagOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java
Test results: passed: 656; failed: 9; error: 1

test-docker-ubuntu2004-armv7l-3 https://ci.adoptium.net/job/Grinder/11460/console

TEST: java/util/PluggableLocale/TimeZoneNameProviderTest.sh
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/CustomFJPoolTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/DoublePrimitiveOpsTests.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/CountTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/ConcatTest.java
TEST: java/util/stream/boottest/java/util/stream/StreamReuseTest.java
TEST: java/util/stream/boottest/java/util/stream/SpinedBufferTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/GroupByOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FindFirstOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/lang/invoke/DeserializeMethodTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FindAnyOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/MatchOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/CountLargeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntPrimitiveOpsTests.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntReduceTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/stream/boottest/java/util/stream/StreamFlagsTest.java
TEST: java/util/stream/boottest/java/util/stream/StreamOpFlagsTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/lang/invoke/MHProxiesTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/LongPrimitiveOpsTests.java
TEST: java/util/stream/boottest/java/util/stream/IntNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/LongNodeTest.java
TEST: java/util/Spliterator/SpliteratorTraversingAndSplittingTest.java
TEST: java/util/stream/boottest/java/util/stream/DoubleNodeTest.java
TEST: java/util/SplittableRandom/SplittableRandomTest.java
TEST: java/util/stream/boottest/java/util/stream/FlagOpTest.java
TEST: java/util/stream/boottest/java/util/stream/SliceSpliteratorTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeBuilderTest.java
TEST: java/util/Random/RandomTest.java
TEST: java/util/ServiceLoader/basic.sh
TEST: java/util/ResourceBundle/Bug4168625Test.java
TEST: java/util/ResourceBundle/Control/BadControlsTest.java
TEST: java/util/concurrent/forkjoin/FJExceptionTableLeak.java
TEST: java/util/concurrent/BlockingQueue/CancelledProducerConsumerLoops.java
TEST: java/util/TreeMap/SubMapClear.java
TEST: java/util/regex/RegExTest.java
TEST: java/util/Map/Collisions.java
TEST: java/util/Map/InPlaceOpsCollisions.java
TEST: java/util/TreeMap/NullPermissiveComparator.java
TEST: java/util/TreeMap/HeadTailTypeError.java
TEST: java/util/TreeMap/NullAtEnd.java
TEST: java/util/TreeMap/EmptyMapAndNulls.java
TEST: java/util/TreeMap/Clone.java
TEST: java/util/TimeZone/TransitionTest.java
TEST: java/util/TimeZone/TimeZoneRegression.java
TEST: java/util/TreeMap/ContainsValue.java
TEST: java/util/TimeZone/SetDefaultSecurityTest.java
TEST: java/util/TimeZone/TimeZoneTest.java
TEST: java/util/TimeZone/UTCAliasTest.java
TEST: java/util/TimeZone/TimeZoneBoundaryTest.java
TEST: java/util/TimeZone/DaylightTimeTest.java
TEST: java/util/TimeZone/HongKong.java
TEST: java/util/TimeZone/TimeZoneDatePermissionCheck.sh
TEST: java/util/TimeZone/CLDRDisplayNamesTest.java
TEST: java/util/TimeZone/OldIDMappingTest.sh
TEST: java/util/TimeZone/ListTimeZones.java
TEST: java/util/TimeZone/IDTest.java
TEST: java/util/concurrent/locks/ReentrantReadWriteLock/MapLoops.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/RangeTest.java
TEST: java/util/zip/ConstructDeflaterInput.java
TEST: java/util/zip/Bounds.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: sun/util/resources/Calendar/Bug4527203.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java
Test results: passed: 595; failed: 3; error: 68

test-docker-debian12-armv7l-1 https://ci.adoptium.net/job/Grinder/11461/console

TEST: java/util/stream/boottest/java/util/stream/DoubleNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/FlagOpTest.java
TEST: java/util/stream/boottest/java/util/stream/IntNodeTest.java
TEST: java/util/stream/boottest/java/util/stream/NodeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/FilterOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntSliceOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/IntUniqOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/RangeTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java
TEST: java/util/concurrent/BlockingQueue/MultipleProducersSingleConsumerLoops.java
Test results: passed: 654; failed: 11; error: 1
sxa commented 1 week ago

Noting that Grinders 4636/4637 in the 'other' jenkins shows similar results - passes on the standalone machine (Pi4) and failing in the container environment on an arm64 host. It might be interesting to try this in a container on a real arm32 host to see if that makes a difference.