jlink is not reproducible

[ReillyBrogan]

I've been trying to get jlink from openj9 to make bit-for-bit reproducible jlink images, however there seems to be an issue specific to openj9 that is causing the resultant images to be non-reproducible. This is easily noticeable with a test-case:

wget https://github.com/ibmruntimes/semeru17-binaries/releases/download/jdk-17.0.7%2B7_openj9-0.38.0/ibm-semeru-open-jdk_x64_linux_17.0.7_7_openj9-0.38.0.tar.gz
tar -xf ibm-semeru-open-jdk_x64_linux_17.0.7_7_openj9-0.38.0.tar.gz
mv jdk-17.0.7+7/ semeru-17

# Create some sample images:
semeru-17/bin/jlink --add-modules java.base,java.compiler --output semeru1
semeru-17/bin/jlink --add-modules java.base,java.compiler --output semeru2

# Normalize modify timestamps
find semeru* -exec touch --date=0 --no-dereference {} \;

# Run diffoscope (https://diffoscope.org/) to show differences:
docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \
                                        registry.salsa.debian.org/reproducible-builds/diffoscope semeru1 semeru2

This returns with an output like the following:

--- semeru1
+++ semeru2
│   --- semeru1/lib
├── +++ semeru2/lib
│ │   --- semeru1/lib/modules
│ ├── +++ semeru2/lib/modules
│ │ @@ -24033,15 +24033,15 @@
│ │  0005de00: 7479 7065 0802 b501 0015 6a61 7661 782e  type......javax.
│ │  0005de10: 6c61 6e67 2e6d 6f64 656c 2e75 7469 6c08  lang.model.util.
│ │  0005de20: 02b7 0100 0b6a 6176 6178 2e74 6f6f 6c73  .....javax.tools
│ │  0005de30: 0802 b901 001d 6a61 7661 782e 746f 6f6c  ......javax.tool
│ │  0005de40: 732e 446f 6375 6d65 6e74 6174 696f 6e54  s.DocumentationT
│ │  0005de50: 6f6f 6c08 02bb 0100 186a 6176 6178 2e74  ool......javax.t
│ │  0005de60: 6f6f 6c73 2e4a 6176 6143 6f6d 7069 6c65  ools.JavaCompile
│ │ +0005de70: 7208 02bd 0300 bb67 a201 000d 6d6f 6475  r......g....modu
│ │ -0005de70: 7208 02bd 0344 09b1 fb01 000d 6d6f 6475  r....D......modu
│ │  0005de80: 6c65 5461 7267 6574 7301 0025 2829 5b4c  leTargets..%()[L
│ │  0005de90: 6a64 6b2f 696e 7465 726e 616c 2f6d 6f64  jdk/internal/mod
│ │  0005dea0: 756c 652f 4d6f 6475 6c65 5461 7267 6574  ule/ModuleTarget
│ │  0005deb0: 3b01 0020 6a64 6b2f 696e 7465 726e 616c  ;.. jdk/internal
│ │  0005dec0: 2f6d 6f64 756c 652f 4d6f 6475 6c65 5461  /module/ModuleTa
│ │  0005ded0: 7267 6574 0702 c201 000b 6c69 6e75 782d  rget......linux-
│ │  0005dee0: 616d 6436 3408 02c4 0a02 c300 1701 000c  amd64...........

Note that if more modules are used the result will be more binary differences in lib/modules.

Hotspot-based JDKs don't exhibit this behavior:

wget https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.7%2B7/OpenJDK17U-jdk_x64_linux_hotspot_17.0.7_7.tar.gz
tar -xf OpenJDK17U-jdk_x64_linux_hotspot_17.0.7_7.tar.gz
mv jdk-17.0.7+7/ temurin-17

# Create some sample images:
temurin-17/bin/jlink --add-modules java.base,java.compiler --output temurin1
temurin-17/bin/jlink --add-modules java.base,java.compiler --output temurin2

# Normalize modify timestamps
find temurin* -exec touch --date=0 --no-dereference {} \;

# Run diffoscope (https://diffoscope.org/) to show differences:
docker run --rm -t -w $(pwd) -v $(pwd):$(pwd):ro \
                                        registry.salsa.debian.org/reproducible-builds/diffoscope temurin1 temurin2

There will be no output which indicates that the two images are identical.

It would be great if this could be resolved so that we can update our build pipelines to be reproducible.

[pshipton]

@tajila fyi

[keithc-ca]

This seems to be due to the computed hash code for ModuleDescriptor objects. Several of the constituents of that computation are enum objects (see ModuleDescriptor.Modifier and ModuleDescriptor.*.Modifier) which inherit hashCode() from java.lang.Enum which just delegates to java.lang.Object.hashCode().

Consider this small test program:

import java.lang.module.ModuleDescriptor;

public class ShowModifiers {

    public static void main(String[] args) {
        show(ModuleDescriptor.Exports.Modifier.class);
        show(ModuleDescriptor.Modifier.class);
        show(ModuleDescriptor.Opens.Modifier.class);
        show(ModuleDescriptor.Requires.Modifier.class);
    }

    private static <T extends Enum<?>> void show(Class<T> cls) {
        System.out.format("%s:%n", cls.getName());

        for (Enum<?> constant : cls.getEnumConstants()) {
            System.out.format("  %-12s -> 0x%08X%n", constant.name(), constant.hashCode());
        }
    }

}

When run with Temurin:

openjdk version "17" 2021-09-14
OpenJDK Runtime Environment Temurin-17+35 (build 17+35)
OpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing)

the output consistently starts with

java.lang.module.ModuleDescriptor$Exports$Modifier:
  SYNTHETIC    -> 0x06B95977
  MANDATED     -> 0x7E9E5F8A

while OpenJ9 yields random hash codes:

java.lang.module.ModuleDescriptor$Exports$Modifier:
  SYNTHETIC    -> 0x86EF27A6
  MANDATED     -> 0xA423EDF5

or

java.lang.module.ModuleDescriptor$Exports$Modifier:
  SYNTHETIC    -> 0x8598B820
  MANDATED     -> 0xFC52C3F0

It seems our choices are either to make hash codes of enum literals consistent across runs, or alter the implementation of ModuleDescriptor.hashCode().

[keithc-ca]

Even though openjdk says it's "Not an issue" or "Won't Fix" it seems they did make enum literal hash codes stable in jdk8u302-b08 or earlier builds.

[keithc-ca]

This is fixed in Java 18; see

• 8275509: ModuleDescriptor.hashCode isn't reproducible across builds

[keithc-ca]

I'll handle this by back-porting the fix to Java 17 and Java 11.

[keithc-ca]

Fixed via the two pull requests referenced above.

[ReillyBrogan]

Not sure if the openj9 test suites use it, but if they do you could probably backport the related JDK-8276688 too.

[keithc-ca]

I believe openj9 uses lists defined in https://github.com/adoptium/aqa-tests in files named ProblemList_openjdkN-openj9.txt. None of those files exclude the JLinkReproducible*Test classes, so I think there's nothing more to be done here. @llxia Do you agree?