search

adoptium / temurin-build

Eclipse Temurin™ build scripts - common across all releases/versions
Apache License 2.0
1.03k stars 251 forks source link

Status of Reproducible comparing tests on linux #3967

Closed sophia-guo closed 1 month ago

sophia-guo commented 2 months ago

PPC64le and aarch64 was enabled on Sep 25th. https://github.com/adoptium/ci-jenkins-pipelines/pull/1115. There is no data yet as no betaTrigger builds triggered with test enabled.

Recent linux update was merged in Sep 6th. .

Most recent ea pipeline jobs are triggered by test disabled (not sure why, for example https://ci.adoptium.net/job/build-scripts/job/openjdk21-pipeline/333/), which means most reproducible comparing tests are quite old and may not be worth looking at. FYI @smlambert @andrew-m-leonard

Only one test build with jdk24 is with recent update. https://trss.adoptium.net/output/test?id=66f72b85d24e1b006ee0dd01 @andrew-m-leonard triggered by https://ci.adoptium.net/job/build-scripts/job/openjdk24-pipeline/25/, test jenkins job is https://ci.adoptium.net/job/Test_openjdk24_hs_special.system_x86-64_linux/11/. Both job and rerun job have same results. 6 files differs.

17:51:56  Number of files: 105604
17:51:56  Comparing jdk-24+17 with tarJDK/jdk-24+17 ... output to file: reprotest.diff
17:52:00  Files jdk-24+17/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-24+17/jmods/expanded_java.base.jmod/lib/classlist differ
17:52:00  Files jdk-24+17/jmods/expanded_java.base.jmod/lib/server/libjvm.so and tarJDK/jdk-24+17/jmods/expanded_java.base.jmod/lib/server/libjvm.so differ
17:52:00  Files jdk-24+17/lib/classlist and tarJDK/jdk-24+17/lib/classlist differ
17:52:00  Files jdk-24+17/lib/server/classes.jsa and tarJDK/jdk-24+17/lib/server/classes.jsa differ
17:52:00  Files jdk-24+17/lib/server/classes_nocoops.jsa and tarJDK/jdk-24+17/lib/server/classes_nocoops.jsa differ
17:52:00  Files jdk-24+17/lib/server/libjvm.so and tarJDK/jdk-24+17/lib/server/libjvm.so differ
17:52:00  Number of differences: 6
17:52:00  ReproduciblePercent = 99.99 %
17:52:00  Differences found..., logged in: reprotest.diff

@andrew-m-leonard I remember you mentioned those CDS files differences ( 4 files) are a known issue due to the classlist ipv6 difference. Compared with the job I tested before https://ci.adoptium.net/job/Test_openjdk23_hs_special.system_x86-64_linux_testList_0/29/console, number of files increased and two more file differs. jmods/expanded_java.base.jmod/lib/server/libjvm.so and /lib/server/libjvm.so and tarJDK/jdk-24+17/lib/server/libjvm.so. Are those two differences expected?

09:59:33  Number of files: 104722
09:59:33  Comparing jdk-23+36 with tarJDK/jdk-23+36 ... output to file: reprotest.diff
09:59:37  Files jdk-23+36/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-23+36/jmods/expanded_java.base.jmod/lib/classlist differ
09:59:37  Files jdk-23+36/lib/classlist and tarJDK/jdk-23+36/lib/classlist differ
09:59:37  Files jdk-23+36/lib/server/classes.jsa and tarJDK/jdk-23+36/lib/server/classes.jsa differ
09:59:37  Files jdk-23+36/lib/server/classes_nocoops.jsa and tarJDK/jdk-23+36/lib/server/classes_nocoops.jsa differ
09:59:37  Number of differences: 4
09:59:37  ReproduciblePercent = 99.99 %
09:59:37  Differences found..., logged in: reprotest.diff``
sophia-guo commented 2 months ago

I see , most of September openjdk pipeline jobs disabled tests job as it's in Release period, which causes only one job with jdk24 available. https://github.com/adoptium/temurin/issues/54#issuecomment-2389545496

andrew-m-leonard commented 2 months ago

@sophia-guo jdk-21.0.5+9 EA is building now.... so you should have a new test to look at soon...

The above differences need looking at, for Linux they should be 100% identical, I suspect an "environmental" difference that we need to investigate in this issue. I suspect one of:

@sophia-guo What would be really useful output in the test, is if the "classlist" file appears in the diff, then do a verbose content diff just for that file, as that will reveal any of the differences suggested above ^

sophia-guo commented 2 months ago

I think you mentioned @andrew-m-leonard 

I suspect you've added the CDS files for Linux due to the classlist ipv6 difference, that is a known issue, and we need to ensure we have the same ipv6 setting on the reproduce VM.
sophia-guo commented 1 month ago

The latest results - linux-x64 https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_x86-64_linux/27/artifact/aqa-tests/TKG/output_17279540359679/Test_openjdk21_hs_special.system_x86-64_linux.tap/*view*/

        Cleaning any Temurin build-scripts release file BUILD_INFO from tarJDK/jdk-21.0.5+9
        Number of files: 108609
        Comparing jdk-21.0.5+9 with tarJDK/jdk-21.0.5+9 ... output to file: reprotest.diff
        Files jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist differ
        Files jdk-21.0.5+9/lib/classlist and tarJDK/jdk-21.0.5+9/lib/classlist differ
        Files jdk-21.0.5+9/lib/server/classes.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes.jsa differ
        Files jdk-21.0.5+9/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes_nocoops.jsa differ
        Number of differences: 4
        ReproduciblePercent = 99.99 %
        Differences found..., logged in: reprotest.diff

linux-aarch64 and ppc64le have the same result https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_ppc64le_linux/1/artifact/aqa-tests/TKG/output_17279518397907/Test_openjdk21_hs_special.system_ppc64le_linux.tap/*view*/

https://ci.adoptium.net/job/Test_openjdk21_hs_special.system_ppc64le_linux/1/artifact/aqa-tests/TKG/output_17279518397907/Test_openjdk21_hs_special.system_ppc64le_linux.tap/*view*/)

        Comparing jdk-21.0.5+9 with tarJDK/jdk-21.0.5+9 ... output to file: reprotest.diff
        Only in jdk-21.0.5+9: jdk-21.0.5+9
        Only in jdk-21.0.5+9: jdk-21.0.5+9-static-libs
        Files jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.5+9/jmods/expanded_java.base.jmod/lib/classlist differ
        Files jdk-21.0.5+9/lib/classlist and tarJDK/jdk-21.0.5+9/lib/classlist differ
        Files jdk-21.0.5+9/lib/server/classes.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes.jsa differ
        Files jdk-21.0.5+9/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.5+9/lib/server/classes_nocoops.jsa differ
        Number of differences: 6
        ReproduciblePercent = 99.99 %
        Differences found..., logged in: reprotest.diff
sophia-guo commented 1 month ago

https://github.com/adoptium/temurin-build/issues/3988

sophia-guo commented 1 month ago

If the "classlist" file appears in the diff, do a verbose content diff just for that file by 'diff -r,' https://github.com/adoptium/temurin-build/compare/master...sophia-guo:openjdk-build:LinuxClass?expand=1

results:

 23:33:55  Cleaning any Temurin build-scripts release file BUILD_INFO from tarJDK/jdk-21.0.4+7
23:33:56  Number of files: 108594
23:33:56  Comparing jdk-21.0.4+7 with tarJDK/jdk-21.0.4+7 ... output to file: reprotest.diff
23:34:00  Files jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist differ
23:34:00  Files jdk-21.0.4+7/lib/classlist and tarJDK/jdk-21.0.4+7/lib/classlist differ
23:34:00  Files jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa differ
23:34:00  Files jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa differ
23:34:00  diff -c on jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist and tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist
23:34:00  *** jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist  Wed Oct 30 03:33:26 2024
23:34:00  --- tarJDK/jdk-21.0.4+7/jmods/expanded_java.base.jmod/lib/classlist   Wed Oct 30 03:33:51 2024
23:34:00  ***************
23:34:00  *** 351,369 ****
23:34:00    java/lang/reflect/Type
23:34:00    java/math/BigInteger
23:34:00    java/math/RoundingMode
23:34:00  - java/net/DefaultInterface
23:34:00    java/net/Inet4Address
23:34:00    java/net/Inet4AddressImpl
23:34:00  - java/net/Inet6Address
23:34:00  - java/net/Inet6Address$Inet6AddressHolder
23:34:00  - java/net/Inet6AddressImpl
23:34:00    java/net/InetAddress
23:34:00    java/net/InetAddress$1
23:34:00    java/net/InetAddress$InetAddressHolder
23:34:00    java/net/InetAddress$PlatformResolver
23:34:00    java/net/InetAddressImpl
23:34:00  - java/net/InterfaceAddress
23:34:00  - java/net/NetworkInterface
23:34:00    java/net/URI
23:34:00    java/net/URI$1
23:34:00    java/net/URI$Parser
23:34:00  --- 351,363 ----
23:34:00  diff -c on jdk-21.0.4+7/lib/classlist and tarJDK/jdk-21.0.4+7/lib/classlist
23:34:00  *** jdk-21.0.4+7/lib/classlist    Wed Oct 30 03:10:54 2024
23:34:00  --- tarJDK/jdk-21.0.4+7/lib/classlist Wed Oct 30 03:30:04 2024
23:34:00  ***************
23:34:00  *** 351,369 ****
23:34:00    java/lang/reflect/Type
23:34:00    java/math/BigInteger
23:34:00    java/math/RoundingMode
23:34:00  - java/net/DefaultInterface
23:34:00    java/net/Inet4Address
23:34:00    java/net/Inet4AddressImpl
23:34:00  - java/net/Inet6Address
23:34:00  - java/net/Inet6Address$Inet6AddressHolder
23:34:00  - java/net/Inet6AddressImpl
23:34:00    java/net/InetAddress
23:34:00    java/net/InetAddress$1
23:34:00    java/net/InetAddress$InetAddressHolder
23:34:00    java/net/InetAddress$PlatformResolver
23:34:00    java/net/InetAddressImpl
23:34:00  - java/net/InterfaceAddress
23:34:00  - java/net/NetworkInterface
23:34:00    java/net/URI
23:34:00    java/net/URI$1
23:34:00    java/net/URI$Parser
23:34:00  --- 351,363 ----
23:34:00  diff -c on jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa
23:34:00  Binary files jdk-21.0.4+7/lib/server/classes.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes.jsa differ
23:34:00  diff -c on jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa
23:34:00  Binary files jdk-21.0.4+7/lib/server/classes_nocoops.jsa and tarJDK/jdk-21.0.4+7/lib/server/classes_nocoops.jsa differ
23:34:00  Number of differences: 4
23:34:00  ReproduciblePercent = 99.99 %
23:34:00  Differences found..., logged in: reprotest.diff
23:34:00  -----------------------------------

https://ci.adoptium.net/job/Grinder/11263/

Using https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-x64-temurin/286/artifact/workspace/target/OpenJDK21U-jdk_x64_linux_hotspot_2024-10-15-21-58.tar.gz tag: 21.0.4+7

sophia-guo commented 1 month ago

@andrew-m-leonard https://github.com/adoptium/temurin-build/issues/3967#issuecomment-2447272791 updated the verbose output.

andrew-m-leonard commented 1 month ago

thanks @sophia-guo so yes the problem is one env has ipv6 enabled the other doesn't... we need to fix one of the environments

andrew-m-leonard commented 1 month ago

We maybe able to dynamically disable it using a sysctl maybe ?

sophia-guo commented 1 month ago

temurin build environment is adoptopenjdk/centos7_build_image and reproducible is uisng centos:7. It seems adoptopenjdk/centos7_build_image IPV6 is enabled while centos:7 isn't? @andrew-m-leonard are you talking about disable it in temurin build?

sophia-guo commented 1 month ago

Google notes

Red-hat based distributions, Change to the root user. Type these commands: disable

sysctl -w net.ipv6.conf.all.disable_ipv6=1 sysctl -w net.ipv6.conf.default.disable_ipv6=1 sysctl -w net.ipv6.conf.tun0.disable_ipv6=1

To re-enable IPv6, type these commands: sysctl -w net.ipv6.conf.all.disable_ipv6=0 sysctl -w net.ipv6.conf.default.disable_ipv6=0 sysctl -w net.ipv6.conf.tun0.disable_ipv6=0 sysctl -p

andrew-m-leonard commented 1 month ago

temurin build environment is adoptopenjdk/centos7_build_image and reproducible is uisng centos:7. It seems adoptopenjdk/centos7_build_image IPV6 is enabled while centos:7 isn't? @andrew-m-leonard are you talking about disable it in temurin build?

@sophia-guo i'm thinking the test script could dynamically enable in the container possibly using sysctl, would need testing to see if it works?

sophia-guo commented 1 month ago

Did a try. Seems this can be done by configure namespaced kernel parameters (sysctls) at runtime (--sysctl). Note Docker does not support changing sysctls inside of a container that also modify the host system. https://docs.docker.com/reference/cli/docker/container/run/#sysctl

Grinder https://ci.adoptium.net/job/Grinder/11270/

andrew-m-leonard commented 1 month ago

Did a try. Seems this can be done by configure namespaced kernel parameters (sysctls) at runtime (--sysctl). Note Docker does not support changing sysctls inside of a container that also modify the host system. https://docs.docker.com/reference/cli/docker/container/run/#sysctl

Grinder https://ci.adoptium.net/job/Grinder/11270/

@sophia-guo Looks good?

19:01:02  ReproduciblePercent = 100 %
19:01:02  Compare identical !
sophia-guo commented 1 month ago

@andrew-m-leonard yes, the PR is here #4020