search

adoptium / infrastructure

This repo contains all information about machine maintenance.
Apache License 2.0
84 stars 100 forks source link

[jdk8 - windows64]: some java_util tests failed with `insufficient memory` #2663

Open sophia-guo opened 2 years ago

sophia-guo commented 2 years ago

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/648/#showFailuresLink

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java.SerializedLambdaTest java/util/stream/test/org/openjdk/tests/java/util/FillableStringTest.java.FillableStringTest java/util/stream/test/org/openjdk/tests/java/util/stream/ConcatOpTest.java.ConcatOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/DistinctOpTest.java.DistinctOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/DoublePrimitiveOpsTests.java.DoublePrimitiveOpsTests java/util/stream/test/org/openjdk/tests/java/util/stream/FindAnyOpTest.java.FindAnyOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/FlatMapOpTest.java.FlatMapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/FlatMapOpTest.java.FlatMapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java.InfiniteStreamWithLimitOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java.InfiniteStreamWithLimitOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java.MapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java.SequentialOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SliceOpTest.java.SliceOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SliceOpTest.java.SliceOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SortedOpTest.java.SortedOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SpliteratorTest.java.SpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java.StreamBuilderTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java.StreamSpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java.StreamSpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/TabulatorsTest.java.TabulatorsTest java/util/stream/test/org/openjdk/tests/java/util/stream/TabulatorsTest.java.TabulatorsTest java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java.TeeOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java.TeeOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/ToArrayOpTest.java.ToArrayOpTest

09:03:34  STDOUT:
09:03:34  #
09:03:34  # There is insufficient memory for the Java Runtime Environment to continue.
09:03:34  # Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap
09:03:34  # An error report file with more information is saved as:
09:03:34  # E:\workspace\Test_openjdk8_hs_sanity.openjdk_x86-64_windows\aqa-tests\TKG\output_16567659847658\jdk_util_1\work\java\util\stream\test\org\openjdk\tests\java\util\stream\TabulatorsTest\hs_err_pid19676.log

Failed on test-azure-win2012r2-x64-3 test-ibmcloud-win2012r2-x64-1.

Haroon-Khel commented 1 year ago

Rerunning

Problem machines: test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5143/console

test-ibmcloud-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5144/console

Other machines: test-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5145/console

test-azure-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5146/console

build-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5147/console

test-azure-win2016-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5148/console

test-azure-win2019-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5149/console

build-azure-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5150/console

build-alibaba-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5151/console

Haroon-Khel commented 1 year ago

Util tests look good. The insufficient memory error seems intermittent. The lang test did not run with the above grinders, i'll re run that one now

Haroon-Khel commented 1 year ago

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java

Problem machines: test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5153/console

test-ibmcloud-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5154/console

Other machines: test-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5145/console

test-azure-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5155/console

build-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5156/console

test-azure-win2016-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5157/console

test-azure-win2019-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5158/console

build-azure-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5159/console

build-alibaba-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5160/console

Haroon-Khel commented 1 year ago

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java looks good too. More evidence that this error is intermittent

sxa commented 1 year ago

Have they failed at all in your recent runs? May be worth trying again on the machines it's failed on but running with 50 iterations or something like that and see if any of them fail :-)

sxa commented 1 year ago

This needs analysis to understand if it's an intermittent failure or not. Bumping to October

sxa commented 1 year ago

@sophia-guo @Haroon-Khel Has this been seen recently?

Haroon-Khel commented 1 year ago

https://ci.adoptium.net/view/Test_grinder/job/Grinder/7120/console jdk_util on test-azure-win2012r2-x64-3 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7121/console java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java on test-azure-win2012r2-x64-3

Haroon-Khel commented 1 year ago

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is not able to connect to jenkins. I can rdp into the machine. From there it looks like its having trouble accessing the internet, for example it cant connect to ci.adoptium.net in a web browser. @AdamBrousseau Are you able to see if test-ibmcloud-win2012r2-x64-1 is having network issues?

Haroon-Khel commented 1 year ago

https://ci.adoptium.net/view/Test_grinder/job/Grinder/7120/console jdk_util on test-azure-win2012r2-x64-3 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7121/console java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java on test-azure-win2012r2-x64-3

Both passed. Just test-ibmcloud-win2012r2-x64-1 left

sxa commented 1 year ago

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is not able to connect to jenkins. I can rdp into the machine. From there it looks like its having trouble accessing the internet, for example it cant connect to ci.adoptium.net in a web browser. @AdamBrousseau Are you able to see if test-ibmcloud-win2012r2-x64-1 is having network issues?

Can it do anything e.g. ssh out? Download something via curl/wget? What errors are we getting on the machine? Does it seem to be hostname resolution for example?

(Clearly the networking isn't fundamentally broken as we can get into it ;-) )

Haroon-Khel commented 1 year ago

curl and wget did not work as it looks like it cannot resolve hostnames, website were not loading in a browser

image image
Haroon-Khel commented 1 year ago

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is back online after manually giving it a dns server (8.8.8.8)

Haroon-Khel commented 1 year ago

jdk_util on test-ibmcloud-1 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7137/console No need for a separate java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java grinder as it runs in jdk_util

Haroon-Khel commented 1 year ago

Out of the 10 iterations in the grinder, these tests were the common failures. Others were intermittent

Failed test cases: 
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamLinkTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/ToArrayOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/FillableStringTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/MapTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/NullArgsTestCase.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/SplittableRandomTest.java
Test results: passed: 662; failed: 8
Haroon-Khel commented 1 year ago

rebuilds

jdk11 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7138/console jdk 17 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7139/console jdk 20 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7140/console

Haroon-Khel commented 1 year ago

Tests pass for jdk 11 17 and 20

adamfarley commented 1 year ago

Heya Haroon. Do you think this could be related?

Haroon-Khel commented 3 months ago

Needs to be rerun on newer machines, alot (if not all) of the original machines are out of commission

sxa commented 3 months ago

Are you suggesting that it would only fail now on win2012 machines or are you using another definition of "newer" there?