search

jvm-repo-rebuild / reproducible-central

Reproducible Central: rebuild instructions for artifacts published to (Maven) Central Repository
102 stars 46 forks source link

Not able to reproduce rebuild issues in Apache Shiro 2.0.1 #185

Closed lprimak closed 3 months ago

lprimak commented 3 months ago

Shiro 2.0.1 is indicating a reproducible build issue. However, when I run reproducible build locally, I cannot "reproduce" this issue, and there are no issue reported in the local RB build.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  07:09 min
[INFO] Finished at: 2024-07-07T00:12:55Z
[INFO] ------------------------------------------------------------------------
dos2unix: converting file target/shiro-root-2.0.1.buildinfo to Unix format...
dos2unix: converting file target/shiro-root-2.0.1.buildcompare to Unix format...

[INFO] rebuild from content/org/apache/shiro/shiro-2.0.1.buildspec
[INFO]   results in content/org/apache/shiro/shiro-root-2.0.1.buildinfo
[INFO] compared to Central Repository content/org/apache/shiro/shiro-root-2.0.1.buildcompare:
    ok=106
    okFiles="shiro-root-2.0.1.pom shiro-root-2.0.1-source-release.zip shiro-lang-2.0.1.pom shiro-lang-2.0.1.jar shiro-lang-2.0.1-sources.jar shiro-crypto-2.0.1.pom shiro-crypto-core-2.0.1.pom shiro-crypto-core-2.0.1.jar shiro-crypto-core-2.0.1-sources.jar shiro-crypto-hash-2.0.1.pom shiro-crypto-hash-2.0.1.jar shiro-crypto-hash-2.0.1-sources.jar shiro-crypto-cipher-2.0.1.pom shiro-crypto-cipher-2.0.1.jar shiro-crypto-cipher-2.0.1-sources.jar shiro-crypto-cipher-2.0.1-jakarta.jar shiro-crypto-support-2.0.1.pom shiro-hashes-argon2-2.0.1.pom shiro-hashes-argon2-2.0.1.jar shiro-hashes-argon2-2.0.1-tests.jar shiro-hashes-argon2-2.0.1-sources.jar shiro-hashes-bcrypt-2.0.1.pom shiro-hashes-bcrypt-2.0.1.jar shiro-hashes-bcrypt-2.0.1-tests.jar shiro-hashes-bcrypt-2.0.1-sources.jar shiro-event-2.0.1.pom shiro-event-2.0.1.jar shiro-event-2.0.1-sources.jar shiro-cache-2.0.1.pom shiro-cache-2.0.1.jar shiro-cache-2.0.1-sources.jar shiro-config-2.0.1.pom shiro-config-core-2.0.1.pom shiro-config-core-2.0.1.jar shiro-config-core-2.0.1-sources.jar shiro-config-ogdl-2.0.1.pom shiro-config-ogdl-2.0.1.jar shiro-config-ogdl-2.0.1-tests.jar shiro-config-ogdl-2.0.1-sources.jar shiro-core-2.0.1.pom shiro-core-2.0.1.jar shiro-core-2.0.1-tests.jar shiro-core-2.0.1-sources.jar shiro-core-2.0.1-jakarta.jar shiro-web-2.0.1.pom shiro-web-2.0.1.jar shiro-web-2.0.1-sources.jar shiro-web-2.0.1-jakarta.jar shiro-support-2.0.1.pom shiro-aspectj-2.0.1.pom shiro-aspectj-2.0.1.jar shiro-aspectj-2.0.1-sources.jar shiro-jcache-2.0.1.pom shiro-jcache-2.0.1.jar shiro-jcache-2.0.1-sources.jar shiro-ehcache-2.0.1.pom shiro-ehcache-2.0.1.jar shiro-ehcache-2.0.1-sources.jar shiro-hazelcast-2.0.1.pom shiro-hazelcast-2.0.1.jar shiro-hazelcast-2.0.1-sources.jar shiro-quartz-2.0.1.pom shiro-quartz-2.0.1.jar shiro-quartz-2.0.1-sources.jar shiro-spring-2.0.1.pom shiro-spring-2.0.1.jar shiro-spring-2.0.1-sources.jar shiro-spring-2.0.1-jakarta.jar shiro-guice-2.0.1.pom shiro-guice-2.0.1.jar shiro-guice-2.0.1-tests.jar shiro-guice-2.0.1-sources.jar shiro-guice-2.0.1-jakarta.jar shiro-spring-boot-2.0.1.pom shiro-spring-boot-starter-2.0.1.pom shiro-spring-boot-starter-2.0.1.jar shiro-spring-boot-starter-2.0.1-sources.jar shiro-spring-boot-starter-2.0.1-jakarta.jar shiro-spring-boot-web-starter-2.0.1.pom shiro-spring-boot-web-starter-2.0.1.jar shiro-spring-boot-web-starter-2.0.1-sources.jar shiro-spring-boot-web-starter-2.0.1-jakarta.jar shiro-servlet-plugin-2.0.1.pom shiro-servlet-plugin-2.0.1.jar shiro-servlet-plugin-2.0.1-sources.jar shiro-jaxrs-2.0.1.pom shiro-jaxrs-2.0.1.jar shiro-jaxrs-2.0.1-sources.jar shiro-jaxrs-2.0.1-jakarta.jar shiro-features-2.0.1.pom shiro-features-2.0.1-features.xml shiro-cdi-2.0.1.pom shiro-cdi-2.0.1.jar shiro-cdi-2.0.1-jakarta.jar shiro-cdi-2.0.1-sources.jar shiro-jakarta-ee-2.0.1.pom shiro-jakarta-ee-2.0.1.jar shiro-jakarta-ee-2.0.1-jakarta.jar shiro-jakarta-ee-2.0.1-sources.jar shiro-tools-2.0.1.pom shiro-tools-hasher-2.0.1.pom shiro-tools-hasher-2.0.1.jar shiro-tools-hasher-2.0.1-sources.jar shiro-tools-hasher-2.0.1-cli.jar shiro-bom-2.0.1.pom shiro-test-coverage-2.0.1.pom"
lprimak@Lennys-MacBook-Pro reproducible-central %

Also, There was another issue as I had to tweak the build files in order to run the build with JDK 22:

includes % git diff
diff --git a/bin/includes/mvnBuildDocker.sh b/bin/includes/mvnBuildDocker.sh
index 2cb4bd528..995ec24b6 100755
--- a/bin/includes/mvnBuildDocker.sh
+++ b/bin/includes/mvnBuildDocker.sh
@@ -54,7 +54,7 @@ mvnBuildDocker() {
       mvnImage=maven:${mvnVersion}-openjdk-${jdk}-slim
       ;;
     *)
-      mvnImage=maven:${mvnVersion}-jdk-${jdk}
+      mvnImage=maven:${mvnVersion}-${jdk}
   esac

   # Second configure it for the local user
diff --git a/content/org/apache/shiro/shiro-2.0.1.buildspec b/content/org/apache/shiro/shiro-2.0.1.buildspec
index d57aff1f5..6e1543307 100644
--- a/content/org/apache/shiro/shiro-2.0.1.buildspec
+++ b/content/org/apache/shiro/shiro-2.0.1.buildspec
@@ -6,8 +6,8 @@ version=2.0.1
 gitRepo=https://github.com/apache/shiro.git
 gitTag=${artifactId}-${version}

-tool=mvn
-jdk=22
+tool=mvn-3.9.8
+jdk=eclipse-temurin-22
 newline=lf
 umask=022

diff --git a/content/org/apache/shiro/shiro-root-2.0.1.buildcompare b/content/org/apache/shiro/shiro-root-2.0.1.buildcompare
index 4b8e63941..0b0af53cb 100644
--- a/content/org/apache/shiro/shiro-root-2.0.1.buildcompare
+++ b/content/org/apache/shiro/shiro-root-2.0.1.buildcompare
@@ -1,10 +1,9 @@
 version=2.0.1
-ok=105
-ko=1
+ok=106
+ko=0
 ignored=0
-okFiles="shiro-root-2.0.1.pom shiro-root-2.0.1-source-release.zip shiro-lang-2.0.1.pom shiro-lang-2.0.1.jar shiro-lang-2.0.1-sources.jar shiro-crypto-2.0.1.pom shiro-crypto-core-2.0.1.pom shiro-crypto-core-2.0.1.jar shiro-crypto-core-2.0.1-sources.jar shiro-crypto-hash-2.0.1.pom shiro-crypto-hash-2.0.1.jar shiro-crypto-hash-2.0.1-sources.jar shiro-crypto-cipher-2.0.1.pom shiro-crypto-cipher-2.0.1.jar shiro-crypto-cipher-2.0.1-sources.jar shiro-crypto-cipher-2.0.1-jakarta.jar shiro-crypto-support-2.0.1.pom shiro-hashes-argon2-2.0.1.pom shiro-hashes-argon2-2.0.1.jar shiro-hashes-argon2-2.0.1-tests.jar shiro-hashes-argon2-2.0.1-sources.jar shiro-hashes-bcrypt-2.0.1.pom shiro-hashes-bcrypt-2.0.1.jar shiro-hashes-bcrypt-2.0.1-tests.jar shiro-hashes-bcrypt-2.0.1-sources.jar shiro-event-2.0.1.pom shiro-event-2.0.1.jar shiro-event-2.0.1-sources.jar shiro-cache-2.0.1.pom shiro-cache-2.0.1.jar shiro-cache-2.0.1-sources.jar shiro-config-2.0.1.pom shiro-config-core-2.0.1.pom shiro-config-core-2.0.1.jar shiro-config-core-2.0.1-sources.jar shiro-config-ogdl-2.0.1.pom shiro-config-ogdl-2.0.1.jar shiro-config-ogdl-2.0.1-tests.jar shiro-config-ogdl-2.0.1-sources.jar shiro-core-2.0.1.pom shiro-core-2.0.1.jar shiro-core-2.0.1-tests.jar shiro-core-2.0.1-sources.jar shiro-core-2.0.1-jakarta.jar shiro-web-2.0.1.pom shiro-web-2.0.1.jar shiro-web-2.0.1-sources.jar shiro-web-2.0.1-jakarta.jar shiro-support-2.0.1.pom shiro-aspectj-2.0.1.pom shiro-aspectj-2.0.1-sources.jar shiro-jcache-2.0.1.pom shiro-jcache-2.0.1.jar shiro-jcache-2.0.1-sources.jar shiro-ehcache-2.0.1.pom shiro-ehcache-2.0.1.jar shiro-ehcache-2.0.1-sources.jar shiro-hazelcast-2.0.1.pom shiro-hazelcast-2.0.1.jar shiro-hazelcast-2.0.1-sources.jar shiro-quartz-2.0.1.pom shiro-quartz-2.0.1.jar shiro-quartz-2.0.1-sources.jar shiro-spring-2.0.1.pom shiro-spring-2.0.1.jar shiro-spring-2.0.1-sources.jar shiro-spring-2.0.1-jakarta.jar shiro-guice-2.0.1.pom shiro-guice-2.0.1.jar shiro-guice-2.0.1-tests.jar shiro-guice-2.0.1-sources.jar shiro-guice-2.0.1-jakarta.jar shiro-spring-boot-2.0.1.pom shiro-spring-boot-starter-2.0.1.pom shiro-spring-boot-starter-2.0.1.jar shiro-spring-boot-starter-2.0.1-sources.jar shiro-spring-boot-starter-2.0.1-jakarta.jar shiro-spring-boot-web-starter-2.0.1.pom shiro-spring-boot-web-starter-2.0.1.jar shiro-spring-boot-web-starter-2.0.1-sources.jar shiro-spring-boot-web-starter-2.0.1-jakarta.jar shiro-servlet-plugin-2.0.1.pom shiro-servlet-plugin-2.0.1.jar shiro-servlet-plugin-2.0.1-sources.jar shiro-jaxrs-2.0.1.pom shiro-jaxrs-2.0.1.jar shiro-jaxrs-2.0.1-sources.jar shiro-jaxrs-2.0.1-jakarta.jar shiro-features-2.0.1.pom shiro-features-2.0.1-features.xml shiro-cdi-2.0.1.pom shiro-cdi-2.0.1.jar shiro-cdi-2.0.1-jakarta.jar shiro-cdi-2.0.1-sources.jar shiro-jakarta-ee-2.0.1.pom shiro-jakarta-ee-2.0.1.jar shiro-jakarta-ee-2.0.1-jakarta.jar shiro-jakarta-ee-2.0.1-sources.jar shiro-tools-2.0.1.pom shiro-tools-hasher-2.0.1.pom shiro-tools-hasher-2.0.1.jar shiro-tools-hasher-2.0.1-sources.jar shiro-tools-hasher-2.0.1-cli.jar shiro-bom-2.0.1.pom shiro-test-coverage-2.0.1.pom"
-koFiles="shiro-aspectj-2.0.1.jar"
+okFiles="shiro-root-2.0.1.pom shiro-root-2.0.1-source-release.zip shiro-lang-2.0.1.pom shiro-lang-2.0.1.jar shiro-lang-2.0.1-sources.jar shiro-crypto-2.0.1.pom shiro-crypto-core-2.0.1.pom shiro-crypto-core-2.0.1.jar shiro-crypto-core-2.0.1-sources.jar shiro-crypto-hash-2.0.1.pom shiro-crypto-hash-2.0.1.jar shiro-crypto-hash-2.0.1-sources.jar shiro-crypto-cipher-2.0.1.pom shiro-crypto-cipher-2.0.1.jar shiro-crypto-cipher-2.0.1-sources.jar shiro-crypto-cipher-2.0.1-jakarta.jar shiro-crypto-support-2.0.1.pom shiro-hashes-argon2-2.0.1.pom shiro-hashes-argon2-2.0.1.jar shiro-hashes-argon2-2.0.1-tests.jar shiro-hashes-argon2-2.0.1-sources.jar shiro-hashes-bcrypt-2.0.1.pom shiro-hashes-bcrypt-2.0.1.jar shiro-hashes-bcrypt-2.0.1-tests.jar shiro-hashes-bcrypt-2.0.1-sources.jar shiro-event-2.0.1.pom shiro-event-2.0.1.jar shiro-event-2.0.1-sources.jar shiro-cache-2.0.1.pom shiro-cache-2.0.1.jar shiro-cache-2.0.1-sources.jar shiro-config-2.0.1.pom shiro-config-core-2.0.1.pom shiro-config-core-2.0.1.jar shiro-config-core-2.0.1-sources.jar shiro-config-ogdl-2.0.1.pom shiro-config-ogdl-2.0.1.jar shiro-config-ogdl-2.0.1-tests.jar shiro-config-ogdl-2.0.1-sources.jar shiro-core-2.0.1.pom shiro-core-2.0.1.jar shiro-core-2.0.1-tests.jar shiro-core-2.0.1-sources.jar shiro-core-2.0.1-jakarta.jar shiro-web-2.0.1.pom shiro-web-2.0.1.jar shiro-web-2.0.1-sources.jar shiro-web-2.0.1-jakarta.jar shiro-support-2.0.1.pom shiro-aspectj-2.0.1.pom shiro-aspectj-2.0.1.jar shiro-aspectj-2.0.1-sources.jar shiro-jcache-2.0.1.pom shiro-jcache-2.0.1.jar shiro-jcache-2.0.1-sources.jar shiro-ehcache-2.0.1.pom shiro-ehcache-2.0.1.jar shiro-ehcache-2.0.1-sources.jar shiro-hazelcast-2.0.1.pom shiro-hazelcast-2.0.1.jar shiro-hazelcast-2.0.1-sources.jar shiro-quartz-2.0.1.pom shiro-quartz-2.0.1.jar shiro-quartz-2.0.1-sources.jar shiro-spring-2.0.1.pom shiro-spring-2.0.1.jar shiro-spring-2.0.1-sources.jar shiro-spring-2.0.1-jakarta.jar shiro-guice-2.0.1.pom shiro-guice-2.0.1.jar shiro-guice-2.0.1-tests.jar shiro-guice-2.0.1-sources.jar shiro-guice-2.0.1-jakarta.jar shiro-spring-boot-2.0.1.pom shiro-spring-boot-starter-2.0.1.pom shiro-spring-boot-starter-2.0.1.jar shiro-spring-boot-starter-2.0.1-sources.jar shiro-spring-boot-starter-2.0.1-jakarta.jar shiro-spring-boot-web-starter-2.0.1.pom shiro-spring-boot-web-starter-2.0.1.jar shiro-spring-boot-web-starter-2.0.1-sources.jar shiro-spring-boot-web-starter-2.0.1-jakarta.jar shiro-servlet-plugin-2.0.1.pom shiro-servlet-plugin-2.0.1.jar shiro-servlet-plugin-2.0.1-sources.jar shiro-jaxrs-2.0.1.pom shiro-jaxrs-2.0.1.jar shiro-jaxrs-2.0.1-sources.jar shiro-jaxrs-2.0.1-jakarta.jar shiro-features-2.0.1.pom shiro-features-2.0.1-features.xml shiro-cdi-2.0.1.pom shiro-cdi-2.0.1.jar shiro-cdi-2.0.1-jakarta.jar shiro-cdi-2.0.1-sources.jar shiro-jakarta-ee-2.0.1.pom shiro-jakarta-ee-2.0.1.jar shiro-jakarta-ee-2.0.1-jakarta.jar shiro-jakarta-ee-2.0.1-sources.jar shiro-tools-2.0.1.pom shiro-tools-hasher-2.0.1.pom shiro-tools-hasher-2.0.1.jar shiro-tools-hasher-2.0.1-sources.jar shiro-tools-hasher-2.0.1-cli.jar shiro-bom-2.0.1.pom shiro-test-coverage-2.0.1.pom"
+koFiles=""
 ignoredFiles=""
 reference_java_version="22 (from MANIFEST.MF Build-Jdk-Spec)"
 reference_os_name="Unix (from pom.properties newline)"
-# diffoscope target/reference/org.apache.shiro/shiro-aspectj-2.0.1.jar support/aspectj/target/shiro-aspectj-2.0.1.jar
diff --git a/content/org/apache/shiro/shiro-root-2.0.1.buildinfo b/content/org/apache/shiro/shiro-root-2.0.1.buildinfo
index eaa379834..72f1befaf 100644
--- a/content/org/apache/shiro/shiro-root-2.0.1.buildinfo
+++ b/content/org/apache/shiro/shiro-root-2.0.1.buildinfo
@@ -310,8 +310,8 @@ outputs.17.0.checksums.sha512=0e73e8e46dce228bd4df3a33739257773179c54aae66ae8034

 outputs.17.1.groupId=org.apache.shiro
 outputs.17.1.filename=shiro-aspectj-2.0.1.jar
-outputs.17.1.length=13778
-outputs.17.1.checksums.sha512=0b19b890809033d0f3292d974933bf51f173c8ac42447ac3effccff80f37096e7524f14cc372ffca248f829dcdad8205d8a2f3685fa95dd58b34131b7b70f6d2
+outputs.17.1.length=13784
+outputs.17.1.checksums.sha512=8c7cf57804e3c499a039e1823d03fd6fbde75adf7b1a01ec362f22033cb991fc1d8dee64793ec6a060b514bb5c2313e5795bd06108f4287f2558ad576014a309

 outputs.17.2.groupId=org.apache.shiro
 outputs.17.2.filename=shiro-aspectj-2.0.1-sources.jar
hboutemy commented 3 months ago

I just rebuilt and got the issue with Aspectj: RB_SHELL=1 ./rebuild.sh content/org/apache/shiro/shiro-2.0.1.buildspec that is described in https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/content/org/apache/shiro/shiro-root-2.0.1.diffoscope

I suppose there is a subtle environment difference that triggers the output difference: let's try to figure out.

I'm on Linux, and my JDK 22 is OpenJDK 64-Bit Server VM Temurin-22.0.1+8 What is your precise env, please?

lprimak commented 3 months ago

Hmmm... I just followed instructions on https://github.com/jvm-repo-rebuild/reproducible-central/blob/master/doc/TOOLS.md

How did you manage to build with JDK 22? I had to do some tweaking. Maybe there is something I am missing?

hboutemy commented 3 months ago

short answer: I ran RB_SHELL=1 ./rebuild.sh content/org/apache/shiro/shiro-2.0.1.buildspec longer answer: this run in interactive mode instead of container, and propose commands to copy/paste, relying on sdkman to switch JDK and Maven versions

I'll try your tweak for getting JDK 22 in container: yes, JDK 22 is a pain to inject because its Docker Hub image name conventions do not match previous ones notice: building releases with non-LTS should be avoided as much as possible, because the pain with getting non-LTS will grow with time

hboutemy commented 3 months ago

fixed by PR #187

I don't know why, but you're right, rebuild output is ok when done inside a container: previously, I rebuilt in interactive mode, and surprisingly, it does not give the same output

I suppose Aspectj has a fragility on environment, but I cannot really explain, but definitively nothing is random: tested both on Mac and Linux, both with Docker and Podman

for Shiro 2.0.1, now it is marked ok as we are able to rebuild consistently: rebuilding in interactive mode is just not ok, without much detail on why...

thanks for your efforts when reporting: it helped me improve JDK 22 support and container support, see #187 for details