search

binaryeq / jcompile

scripts to compile Java projects with different compilers to create a data set of comparable binaries
Apache License 2.0
0 stars 0 forks source link

1/3 compilations fail on `wtwhite-vuw-vm` #6

Open wtwhite opened 1 year ago

wtwhite commented 1 year ago

537 of 795 compilation attempts succeed in about 50 minutes:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ /usr/bin/time ./docker-build-all.sh
--snip--
-------------------------------------
2977.60user 241.46system 2:57:03elapsed 30%CPU (0avgtext+0avgdata 364756maxresident)k
0inputs+6986608outputs (6697major+52440847minor)pagefaults 0swaps
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ find jars -type f|sed -Ee 's/.*\.//'|sort|uniq -c
    258 error
    537 jar
wtwhite commented 1 year ago

Puzzled that errors and don't correspond exactly to BUILD FAILURE in logs:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ find jars -type f|sed -Ee 's/.*\.//'|sort|uniq -c
    258 error
    537 jar
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ echo $((258+537))
795
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls build.*.log|wc -l
795
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep 'BUILD FAILURE' build.*.log|wc -l
183
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep 'BUILD SUCCESS' build.*.log|wc -l
612

Indeed, some BUILD SUCCESSes get turned into .jar.error files anyway:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep 'BUILD SUCCESS' jars/*/*.jar.error|wc -l
75

Will look into these in more detail.

wtwhite commented 1 year ago

All 75 are variants of the bcel project:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep 'BUILD SUCCESS' jars/*/*.jar.error
jars/openjdk-10.0.2/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-10.0.2/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-10.0.2/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-10.0.2/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-10.0.2/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.12/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.12/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.12/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.12/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.12/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.16/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.16/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.16/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.16/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.16/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.19/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.19/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.19/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.19/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-11.0.19/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-15.0.1/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-15.0.1/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-15.0.1/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-15.0.1/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-15.0.1/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-16.0.1/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-16.0.1/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-16.0.1/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-16.0.1/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-16.0.1/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.1/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.1/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.1/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.1/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.1/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.4/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.4/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.4/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.4/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.4/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.7/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.7/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.7/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.7/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-17.0.7/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-18.0.2/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-18.0.2/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-18.0.2/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-18.0.2/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-18.0.2/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-19.0.2/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-19.0.2/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-19.0.2/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-19.0.2/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-19.0.2/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.302/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.302/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.302/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.302/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.302/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.342/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.342/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.342/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.342/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.342/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.372/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.372/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.372/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.372/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-8.0.372/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-9.0.1/bcel-6.4.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-9.0.1/bcel-6.4.1.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-9.0.1/bcel-6.5.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-9.0.1/bcel-6.6.0.jar.error:[INFO] BUILD SUCCESS
jars/openjdk-9.0.1/bcel-6.6.1.jar.error:[INFO] BUILD SUCCESS
wtwhite commented 1 year ago

Some but not all of these errors are caused by the HEAD pointing to a raw commit instead of a branch or tag:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep 'Detecting the current branch failed' jars/*/*.jar.error
jars/openjdk-10.0.2/bcel-6.4.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/bcel-6.4.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/bcel-6.5.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/bcel-6.6.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/bcel-6.6.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.10.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.11.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.12.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.8.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.8.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-10.0.2/commons-lang3-3.9.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/bcel-6.4.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/bcel-6.4.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/bcel-6.5.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/bcel-6.6.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/bcel-6.6.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.10.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.11.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.12.0.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.8.1.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.8.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
jars/openjdk-9.0.1/commons-lang3-3.9.jar.error:Detecting the current branch failed: fatal: ref HEAD is not a symbolic ref
wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel/.git$ cat HEAD 
bbaf623d750f030d186ed026dc201995145c63ec

git thinks there are a whole bunch of changes:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel$ git status
HEAD detached at rel/commons-bcel-6.4.0
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   .github/workflows/maven.yml
    deleted:    .travis.yml
    modified:   CONTRIBUTING.md
    modified:   NOTICE.txt
    modified:   README.md
    modified:   RELEASE-NOTES.txt
    modified:   TODO.JustIce
    modified:   docs/verifier/GNUmakefile
    modified:   docs/verifier/JustIce.lyx
    modified:   pom.xml
    modified:   src/changes/changes.xml
    modified:   src/conf/checkstyle-suppressions.xml
--snip--
    modified:   src/test/java/org/apache/bcel/generic/InstructionHandleTestCase.java
    modified:   src/test/java/org/apache/bcel/generic/JdkGenericDumpTestCase.java
    deleted:    src/test/java/org/apache/bcel/visitors/CounterVisitor.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .DS_Store
    .github/GH-ROBOTS.txt
    .github/dependabot.yml
    .github/workflows/codeql-analysis.yml
    .github/workflows/coverage.yml
    .github/workflows/scorecards-analysis.yml
    ?/
    CODE_OF_CONDUCT.md
    SECURITY.md
    src/conf/pmd-ruleset.xml
    src/conf/spotbugs-exclude-filter.xml
--snip--
    src/test/java/org/apache/bcel/generic/LSHRTestCase.java
    src/test/java/org/apache/bcel/generic/LSUBTestCase.java
    src/test/java/org/apache/bcel/generic/LXORTestCase.java

no changes added to commit (use "git add" and/or "git commit -a")

And this blocks the attempt to run git checkout:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel$ git checkout rel/commons-bcel-6.5.0
error: Your local changes to the following files would be overwritten by checkout:
    .github/workflows/maven.yml
    NOTICE.txt
    README.md
    RELEASE-NOTES.txt
    TODO.JustIce
    docs/verifier/GNUmakefile
    docs/verifier/JustIce.lyx
    pom.xml
    src/changes/changes.xml
    src/conf/checkstyle.xml
    src/examples/ClassDumper.java
    src/examples/HelloWorldBuilder.java
--snip--
    src/main/java/org/apache/bcel/classfile/Utility.java
    src/main/java/org/apache/bcel/classfile/package.html
    src/main/java/org/apache/bcel/generic/AnnotationElementValueGen.java
    src/main/java
error: The following untracked working tree files would be overwritten by checkout:
    src/test/java/org/apache/bcel/classfile/CodeExceptionTestCase.java
    src/test/java/org/apache/bcel/classfile/LocalVariableTestCase.java
    src/test/java/org/apache/bcel/generic/EmptyStaticInit.java
Please move or remove them before you switch branches.
Aborting

After that checkout failure, what probably happens is that whatever version of the project was already there gets (re)built, producing a jar with the wrong version number, which makes docker-build-project.sh think the script has failed (which it actually has):

wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel$ git log -1
commit bbaf623d750f030d186ed026dc201995145c63ec (HEAD, tag: rel/commons-bcel-6.4.0, tag: commons-bcel-6.4.0-RC2)
Author: Gary Gregory <gardgregory@gmail.com>
Date:   Fri Sep 20 15:08:21 2019 -0400

    Update POM version numbers for Apache Commons BCEL release 6.4.0
wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel$ ls target/
antrun  apidocs  bcel-6.7.0.jar  bcel-6.7.0-sources.jar  bcel-6.7.0-test-sources.jar  classes  generated-sources  maven-archiver  maven-status  osgi
wtwhite@wtwhite-vuw-vm:~/code/jcompile/dataset/commons-bcel$ ls -ltr target/*.jar
-rw-r--r-- 1 wtwhite wtwhite 718804 Oct 18 15:42 target/bcel-6.7.0.jar
-rw-r--r-- 1 wtwhite wtwhite 694036 Oct 18 15:42 target/bcel-6.7.0-sources.jar
-rw-r--r-- 1 wtwhite wtwhite 182801 Oct 18 15:42 target/bcel-6.7.0-test-sources.jar

I think this means that every jar that was successfully placed in jars/ was built from the right version of the source, so we don't have to rerun any of them.

This also makes me realise that this workflow isn't parallelisable yet -- we need to check out each different tag of a repo into a separate worktree.

wtwhite commented 1 year ago

Starting over with a freshly unzipped dataset/ folder, it this problem affects bcel and commons-lang:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/FRESH/dataset$ for d in */; do echo "$d `git -C $d status|wc -l`"; done
commons-bcel/ 606
commons-codec/ 2
commons-compress/ 6
commons-configuration/ 6
commons-csv/ 6
commons-io/ 8
commons-lang/ 52
commons-net/ 6
guava/ 2
jackson-core/ 6
JSON-java/ 6
wtwhite@wtwhite-vuw-vm:~/code/jcompile/FRESH/dataset$ cd commons-lang/
wtwhite@wtwhite-vuw-vm:~/code/jcompile/FRESH/dataset/commons-lang$ git status
HEAD detached at LANG_3_8_1
Changes not staged for commit:
  (use "git add/rm <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   .travis.yml
    modified:   README.md
    modified:   RELEASE-NOTES.txt
    modified:   checkstyle.xml
    deleted:    findbugs-exclude-filter.xml
    modified:   pom.xml
    modified:   src/assembly/src.xml
    modified:   src/changes/changes.xml
    modified:   src/changes/release-notes.vm
    modified:   src/main/java/org/apache/commons/lang3/ArrayUtils.java
    modified:   src/main/java/org/apache/commons/lang3/CharSet.java
    modified:   src/main/java/org/apache/commons/lang3/ClassUtils.java
    modified:   src/main/java/org/apache/commons/lang3/Conversion.java
    modified:   src/main/java/org/apache/commons/lang3/JavaVersion.java
    modified:   src/main/java/org/apache/commons/lang3/LocaleUtils.java
    modified:   src/main/java/org/apache/commons/lang3/ObjectUtils.java
    modified:   src/main/java/org/apache/commons/lang3/RandomStringUtils.java
    modified:   src/main/java/org/apache/commons/lang3/RandomUtils.java
    modified:   src/main/java/org/apache/commons/lang3/SerializationUtils.java
    modified:   src/main/java/org/apache/commons/lang3/StringEscapeUtils.java
    modified:   src/main/java/org/apache/commons/lang3/StringUtils.java
    modified:   src/main/java/org/apache/commons/lang3/SystemUtils.java
    modified:   src/main/java/org/apache/commons/lang3/ThreadUtils.java
    modified:   src/main/java/org/apache/commons/lang3/builder/Diff.java
    modified:   src/main/java/org/apache/commons/lang3/builder/DiffBuilder.java
    modified:   src/main/java/org/apache/commons/lang3/builder/DiffResult.java
    modified:   src/main/java/org/apache/commons/lang3/builder/Diffable.java
    modified:   src/main/java/org/apache/commons/lang3/builder/EqualsBuilder.java
    modified:   src/main/java/org/apache/commons/lang3/builder/ReflectionDiffBuilder.java
    modified:   src/main/java/org/apache/commons/lang3/builder/ToStringStyle.java
    modified:   src/main/java/org/apache/commons/lang3/concurrent/TimedSemaphore.java
    modified:   src/main/java/org/apache/commons/lang3/exception/ContextedException.java
    modified:   src/main/java/org/apache/commons/lang3/exception/ContextedRuntimeException.java
    modified:   src/main/java/org/apache/commons/lang3/exception/DefaultExceptionContext.java
    modified:   src/main/java/org/apache/commons/lang3/exception/ExceptionUtils.java
    deleted:    src/site/resources/profile.cobertura
    deleted:    src/test/java/org/apache/commons/lang3/test/SystemDefaults.java
    deleted:    src/test/java/org/apache/commons/lang3/test/SystemDefaultsSwitch.java
    deleted:    src/test/java/org/apache/commons/lang3/test/SystemDefaultsSwitchTest.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .DS_Store
    Jenkinsfile
    spotbugs-exclude-filter.xml
    src/main/java/org/apache/commons/lang3/Functions.java

no changes added to commit (use "git add" and/or "git commit -a")

Just adding --force to the git checkout isn't enough, since (a) it won't eliminate existing untracked files, which Maven might continue to pick up, and (b) as mentioned in the previous message, trying to switch between different tags in the same worktree prevents parallelisation.

wtwhite commented 1 year ago

After merging #8, we get a lot more jars successfully built, though by no means all:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ find jars -type f|sed -Ee 's/.*\.//'|sort|uniq -c
    119 error
    676 jar

All bcel projects now build, except for old versions with new JDKs:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls jars/*/bcel-*.*|wc -l
90
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls jars/*/bcel-*.jar|wc -l
76
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls jars/*/bcel-*.jar.error|wc -l
14
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ ls jars/*/bcel-*.jar.error
jars/openjdk-15.0.1/bcel-6.4.0.jar.error  jars/openjdk-16.0.1/bcel-6.4.1.jar.error  jars/openjdk-17.0.4/bcel-6.4.0.jar.error  jars/openjdk-17.0.7/bcel-6.4.1.jar.error  jars/openjdk-19.0.2/bcel-6.4.0.jar.error
jars/openjdk-15.0.1/bcel-6.4.1.jar.error  jars/openjdk-17.0.1/bcel-6.4.0.jar.error  jars/openjdk-17.0.4/bcel-6.4.1.jar.error  jars/openjdk-18.0.2/bcel-6.4.0.jar.error  jars/openjdk-19.0.2/bcel-6.4.1.jar.error
jars/openjdk-16.0.1/bcel-6.4.0.jar.error  jars/openjdk-17.0.1/bcel-6.4.1.jar.error  jars/openjdk-17.0.7/bcel-6.4.0.jar.error  jars/openjdk-18.0.2/bcel-6.4.1.jar.error
wtwhite commented 1 year ago

Over half of the remaining build failures are caused by a ConcurrentModificationException:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -c ConcurrentModificationException jars/*/*.jar.error|grep -v ':0$'|wc -l
63
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -C 3 ConcurrentModificationException jars/*/*.jar.error|head -20
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[INFO] 
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[INFO] --- bundle:4.2.1:manifest (bundle-manifest) @ bcel ---
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[ERROR] An internal error occurred
jars/openjdk-15.0.1/bcel-6.4.0.jar.error:java.util.ConcurrentModificationException
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-    at java.util.TreeMap.callMappingFunctionWithCheck (TreeMap.java:742)
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-    at java.util.TreeMap.computeIfAbsent (TreeMap.java:558)
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-    at aQute.bnd.osgi.Jar.putResource (Jar.java:288)
--
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[WARNING] 
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[WARNING] For more or less details, use 'maven.plugin.validation' property with one of the values (case insensitive): [BRIEF, DEFAULT, VERBOSE]
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[WARNING] 
jars/openjdk-15.0.1/bcel-6.4.0.jar.error:[ERROR] Failed to execute goal org.apache.felix:maven-bundle-plugin:4.2.1:manifest (bundle-manifest) on project bcel: Internal error in maven-bundle-plugin: ConcurrentModificationException -> [Help 1]
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[ERROR] 
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
jars/openjdk-15.0.1/bcel-6.4.0.jar.error-[ERROR] Re-run Maven using the -X switch to enable full debug logging.
--
jars/openjdk-15.0.1/bcel-6.4.1.jar.error-[INFO] 
jars/openjdk-15.0.1/bcel-6.4.1.jar.error-[INFO] --- bundle:4.2.1:manifest (bundle-manifest) @ bcel ---
jars/openjdk-15.0.1/bcel-6.4.1.jar.error-[ERROR] An internal error occurred
jars/openjdk-15.0.1/bcel-6.4.1.jar.error:java.util.ConcurrentModificationException

All of these occur on JDK >= 15.0.1:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l ConcurrentModificationException jars/*/*.jar.error
jars/openjdk-15.0.1/bcel-6.4.0.jar.error
jars/openjdk-15.0.1/bcel-6.4.1.jar.error
jars/openjdk-15.0.1/commons-configuration2-2.6.jar.error
jars/openjdk-15.0.1/commons-configuration2-2.7.jar.error
jars/openjdk-15.0.1/commons-csv-1.7.jar.error
jars/openjdk-15.0.1/commons-csv-1.8.jar.error
jars/openjdk-15.0.1/commons-io-2.7.jar.error
jars/openjdk-15.0.1/commons-lang3-3.10.jar.error
jars/openjdk-15.0.1/commons-lang3-3.9.jar.error
jars/openjdk-16.0.1/bcel-6.4.0.jar.error
jars/openjdk-16.0.1/bcel-6.4.1.jar.error
jars/openjdk-16.0.1/commons-configuration2-2.6.jar.error
jars/openjdk-16.0.1/commons-configuration2-2.7.jar.error
jars/openjdk-16.0.1/commons-csv-1.7.jar.error
jars/openjdk-16.0.1/commons-csv-1.8.jar.error
jars/openjdk-16.0.1/commons-io-2.7.jar.error
jars/openjdk-16.0.1/commons-lang3-3.10.jar.error
jars/openjdk-16.0.1/commons-lang3-3.9.jar.error
jars/openjdk-17.0.1/bcel-6.4.0.jar.error
jars/openjdk-17.0.1/bcel-6.4.1.jar.error
jars/openjdk-17.0.1/commons-configuration2-2.6.jar.error
jars/openjdk-17.0.1/commons-configuration2-2.7.jar.error
jars/openjdk-17.0.1/commons-csv-1.7.jar.error
jars/openjdk-17.0.1/commons-csv-1.8.jar.error
jars/openjdk-17.0.1/commons-io-2.7.jar.error
jars/openjdk-17.0.1/commons-lang3-3.10.jar.error
jars/openjdk-17.0.1/commons-lang3-3.9.jar.error
jars/openjdk-17.0.4/bcel-6.4.0.jar.error
jars/openjdk-17.0.4/bcel-6.4.1.jar.error
jars/openjdk-17.0.4/commons-configuration2-2.6.jar.error
jars/openjdk-17.0.4/commons-configuration2-2.7.jar.error
jars/openjdk-17.0.4/commons-csv-1.7.jar.error
jars/openjdk-17.0.4/commons-csv-1.8.jar.error
jars/openjdk-17.0.4/commons-io-2.7.jar.error
jars/openjdk-17.0.4/commons-lang3-3.10.jar.error
jars/openjdk-17.0.4/commons-lang3-3.9.jar.error
jars/openjdk-17.0.7/bcel-6.4.0.jar.error
jars/openjdk-17.0.7/bcel-6.4.1.jar.error
jars/openjdk-17.0.7/commons-configuration2-2.6.jar.error
jars/openjdk-17.0.7/commons-configuration2-2.7.jar.error
jars/openjdk-17.0.7/commons-csv-1.7.jar.error
jars/openjdk-17.0.7/commons-csv-1.8.jar.error
jars/openjdk-17.0.7/commons-io-2.7.jar.error
jars/openjdk-17.0.7/commons-lang3-3.10.jar.error
jars/openjdk-17.0.7/commons-lang3-3.9.jar.error
jars/openjdk-18.0.2/bcel-6.4.0.jar.error
jars/openjdk-18.0.2/bcel-6.4.1.jar.error
jars/openjdk-18.0.2/commons-configuration2-2.6.jar.error
jars/openjdk-18.0.2/commons-configuration2-2.7.jar.error
jars/openjdk-18.0.2/commons-csv-1.7.jar.error
jars/openjdk-18.0.2/commons-csv-1.8.jar.error
jars/openjdk-18.0.2/commons-io-2.7.jar.error
jars/openjdk-18.0.2/commons-lang3-3.10.jar.error
jars/openjdk-18.0.2/commons-lang3-3.9.jar.error
jars/openjdk-19.0.2/bcel-6.4.0.jar.error
jars/openjdk-19.0.2/bcel-6.4.1.jar.error
jars/openjdk-19.0.2/commons-configuration2-2.6.jar.error
jars/openjdk-19.0.2/commons-configuration2-2.7.jar.error
jars/openjdk-19.0.2/commons-csv-1.7.jar.error
jars/openjdk-19.0.2/commons-csv-1.8.jar.error
jars/openjdk-19.0.2/commons-io-2.7.jar.error
jars/openjdk-19.0.2/commons-lang3-3.10.jar.error
jars/openjdk-19.0.2/commons-lang3-3.9.jar.error
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l ConcurrentModificationException jars/*/*.jar.error|cut -d/ -f2|sort|uniq -c
      9 openjdk-15.0.1
      9 openjdk-16.0.1
      9 openjdk-17.0.1
      9 openjdk-17.0.4
      9 openjdk-17.0.7
      9 openjdk-18.0.2
      9 openjdk-19.0.2
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l ConcurrentModificationException jars/*/*.jar.error|cut -d/ -f3|sort|uniq -c
      7 bcel-6.4.0.jar.error
      7 bcel-6.4.1.jar.error
      7 commons-configuration2-2.6.jar.error
      7 commons-configuration2-2.7.jar.error
      7 commons-csv-1.7.jar.error
      7 commons-csv-1.8.jar.error
      7 commons-io-2.7.jar.error
      7 commons-lang3-3.10.jar.error
      7 commons-lang3-3.9.jar.error
wtwhite commented 1 year ago

This SO answer indicates we need to upgrade maven-bundle-plugin.

We wind up using version 4.2.1, but it's not clear where that comes from -- maven-bundle-plugin is not mentioned explicitly in the pom.xml:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ grep -A 6 maven-bundle-plugin pom.xml 
    <!-- Configuration properties for the OSGi maven-bundle-plugin -->
    <commons.osgi.symbolicName>org.apache.${commons.componentid}</commons.osgi.symbolicName>
    <commons.osgi.export>org.apache.bcel.*;version=${project.version};-noimport:=true</commons.osgi.export>
    <commons.osgi.import>*</commons.osgi.import>
    <commons.osgi.dynamicImport />
    <commons.osgi.private />
    <commons.jira.id>BCEL</commons.jira.id>

And the parent POM mentions the plugin twice, only giving the version in one place -- but via a property:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ grep -C 2 maven-bundle-plugin ~/code/jcompile/.m2/org/apache/commons/commons-parent/49/commons-parent-49.pom
    <commons.packageId>${project.artifactId}</commons.packageId>

    <!-- Configuration properties for the OSGi maven-bundle-plugin -->
    <commons.osgi.symbolicName>org.apache.commons.${commons.packageId}</commons.osgi.symbolicName>
    <commons.osgi.export>org.apache.commons.*;version=${project.version};-noimport:=true</commons.osgi.export>
--
        <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>${commons.felix.version}</version>
          <inherited>true</inherited>
--
      <plugin>
        <groupId>org.apache.felix</groupId>
        <artifactId>maven-bundle-plugin</artifactId>
        <configuration>
          <!--

Finally we find where the version 4.2.1 comes from (the bottom line turns out to be for a different profile):

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ grep -F commons.felix.version !$
grep -F commons.felix.version ~/code/jcompile/.m2/org/apache/commons/commons-parent/49/commons-parent-49.pom
    <commons.felix.version>4.2.1</commons.felix.version>
          <version>${commons.felix.version}</version>
        <commons.felix.version>3.5.1</commons.felix.version>
wtwhite commented 1 year ago

At rel/commons-bcel-6.4.0, under JDK17:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ java --version
openjdk 17.0.8.1 2023-08-24
OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu122.04)
OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu122.04, mixed mode, sharing)
wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ git log -1
commit bbaf623d750f030d186ed026dc201995145c63ec (HEAD, tag: rel/commons-bcel-6.4.0, tag: commons-bcel-6.4.0-RC2)
Author: Gary Gregory <gardgregory@gmail.com>
Date:   Fri Sep 20 15:08:21 2019 -0400

    Update POM version numbers for Apache Commons BCEL release 6.4.0

The default mvn command line fails:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ mvn -Dmaven.repo.local=/home/wtwhite/code/jcompile/.m2 -Drat.skip=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dcyclonedx.skip=true clean package
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------< org.apache.bcel:bcel >------------------------
[INFO] Building Apache Commons BCEL 6.4.0
--snip--
[INFO] --- maven-bundle-plugin:4.2.1:manifest (bundle-manifest) @ bcel ---
[ERROR] An internal error occurred
java.util.ConcurrentModificationException
    at java.util.TreeMap.callMappingFunctionWithCheck (TreeMap.java:750)
    at java.util.TreeMap.computeIfAbsent (TreeMap.java:558)
    at aQute.bnd.osgi.Jar.putResource (Jar.java:288)
    at aQute.bnd.osgi.Jar$1.visitFile (Jar.java:202)
    at aQute.bnd.osgi.Jar$1.visitFile (Jar.java:177)
    at java.nio.file.Files.walkFileTree (Files.java:2811)
    at aQute.bnd.osgi.Jar.buildFromDirectory (Jar.java:176)
    at aQute.bnd.osgi.Jar.<init> (Jar.java:119)
    at aQute.bnd.osgi.Jar.<init> (Jar.java:172)
    at org.apache.felix.bundleplugin.BundlePlugin.getOSGiBuilder (BundlePlugin.java:604)
    at org.apache.felix.bundleplugin.ManifestPlugin.getAnalyzer (ManifestPlugin.java:285)
    at org.apache.felix.bundleplugin.ManifestPlugin.execute (ManifestPlugin.java:111)
    at org.apache.felix.bundleplugin.BundlePlugin.execute (BundlePlugin.java:364)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:77)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:568)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.993 s
[INFO] Finished at: 2023-10-19T17:18:51+13:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.felix:maven-bundle-plugin:4.2.1:manifest (bundle-manifest) on project bcel: Internal error in maven-bundle-plugin: ConcurrentModificationException -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

But overriding the plugin version with a later one via -Dcommons.felix.version=5.1.1 on the mvn command line causes the build to succeed:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ mvn -Dcommons.felix.version=5.1.1 -Dmaven.repo.local=/home/wtwhite/code/jcompile/.m2 -Drat.skip=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dcyclonedx.skip=true clean package
[INFO] Scanning for projects...
[INFO] 
[INFO] ------------------------< org.apache.bcel:bcel >------------------------
[INFO] Building Apache Commons BCEL 6.4.0
--snip--
[INFO] --- maven-bundle-plugin:5.1.1:manifest (bundle-manifest) @ bcel ---
--snip--
[INFO] --- maven-source-plugin:3.1.0:jar (attach-sources) @ bcel ---
[INFO] Building jar: /home/wtwhite/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM/target/bcel-6.4.0-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  5.371 s
[INFO] Finished at: 2023-10-19T17:54:58+13:00
[INFO] ------------------------------------------------------------------------
wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-15.0.1_bcel-6.4.0_MANUALLY_BY_TIM$ ls -ltr target/*.jar
-rw-rw-r-- 1 wtwhite wtwhite 697270 Oct 19 17:54 target/bcel-6.4.0.jar
-rw-rw-r-- 1 wtwhite wtwhite 105090 Oct 19 17:54 target/bcel-6.4.0-test-sources.jar
-rw-rw-r-- 1 wtwhite wtwhite 695181 Oct 19 17:54 target/bcel-6.4.0-sources.jar

Question: Should we force this override some or all of the time?

wtwhite commented 1 year ago

9 considers that question further. In this issue, I'll continue investigating the other 119-63=56 build failures.

wtwhite commented 1 year ago

The remaining 56 build failures again affect the outer product of the same subset of more recent JDK versions with a handful of specific projects:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -c ConcurrentModificationException jars/*/*.jar.error|grep ':0$'|cut -d/ -f2|sort|uniq -c
      8 openjdk-15.0.1
      8 openjdk-16.0.1
      8 openjdk-17.0.1
      8 openjdk-17.0.4
      8 openjdk-17.0.7
      8 openjdk-18.0.2
      8 openjdk-19.0.2
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -c ConcurrentModificationException jars/*/*.jar.error|grep ':0$'|cut -d/ -f3|sort|uniq -c
      7 commons-codec-1.11.jar.error:0
      7 commons-compress-1.16.1.jar.error:0
      7 commons-compress-1.16.jar.error:0
      7 json-20211205.jar.error:0
      7 json-20220320.jar.error:0
      7 json-20220924.jar.error:0
      7 json-20230227.jar.error:0
      7 json-20230618.jar.error:0

Of these 56, 42 are caused by too-old versions being specified for maven.compiler.source or maven.compiler.target:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l 'Source option 6 is no longer supported' `cat remaining_build_failures.txt`|wc -l
42
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l 'Source option 6 is no longer supported' `cat remaining_build_failures.txt`|cut -d/ -f2|sort|uniq -c
      6 openjdk-15.0.1
      6 openjdk-16.0.1
      6 openjdk-17.0.1
      6 openjdk-17.0.4
      6 openjdk-17.0.7
      6 openjdk-18.0.2
      6 openjdk-19.0.2
wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -l 'Source option 6 is no longer supported' `cat remaining_build_failures.txt`|cut -d/ -f3|sort|uniq -c
      7 commons-codec-1.11.jar.error
      7 json-20211205.jar.error
      7 json-20220320.jar.error
      7 json-20220924.jar.error
      7 json-20230227.jar.error
      7 json-20230618.jar.error

These could probably be fixed by specifying -Dmaven.compiler.source=7 -Dmaven.compiler.target=7 to mvn.

wtwhite commented 1 year ago

That leaves 14 remaining build failures, all involving the 2 oldest versions of commons-compress with the latest 7 compiler versions:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -L 'Source option 6 is no longer supported' `cat remaining_build_failures.txt`
jars/openjdk-15.0.1/commons-compress-1.16.1.jar.error
jars/openjdk-15.0.1/commons-compress-1.16.jar.error
jars/openjdk-16.0.1/commons-compress-1.16.1.jar.error
jars/openjdk-16.0.1/commons-compress-1.16.jar.error
jars/openjdk-17.0.1/commons-compress-1.16.1.jar.error
jars/openjdk-17.0.1/commons-compress-1.16.jar.error
jars/openjdk-17.0.4/commons-compress-1.16.1.jar.error
jars/openjdk-17.0.4/commons-compress-1.16.jar.error
jars/openjdk-17.0.7/commons-compress-1.16.1.jar.error
jars/openjdk-17.0.7/commons-compress-1.16.jar.error
jars/openjdk-18.0.2/commons-compress-1.16.1.jar.error
jars/openjdk-18.0.2/commons-compress-1.16.jar.error
jars/openjdk-19.0.2/commons-compress-1.16.1.jar.error
jars/openjdk-19.0.2/commons-compress-1.16.jar.error

Example failure:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ cat !$
cat jars/openjdk-15.0.1/commons-compress-1.16.1.jar.error
[INFO] Scanning for projects...
[INFO] 
[INFO] ----------------< org.apache.commons:commons-compress >-----------------
[INFO] Building Apache Commons Compress 1.16.1
--snip--
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorInputStream.java:[28,21] cannot find symbol
  symbol:   class Pack200
  location: package java.util.jar
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[29,21] cannot find symbol
  symbol:   class Pack200
  location: package java.util.jar
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorOutputStream.java:[26,21] cannot find symbol
  symbol:   class Pack200
  location: package java.util.jar
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorInputStream.java:[175,26] package Pack200 does not exist
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorInputStream.java:[175,40] cannot find symbol
  symbol:   variable Pack200
  location: class org.apache.commons.compress.compressors.pack200.Pack200CompressorInputStream
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[132,26] package Pack200 does not exist
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[137,30] package Pack200 does not exist
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[137,47] cannot find symbol
  symbol:   variable Pack200
  location: class org.apache.commons.compress.compressors.pack200.Pack200Utils
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[141,26] package Pack200 does not exist
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200Utils.java:[141,47] cannot find symbol
  symbol:   variable Pack200
  location: class org.apache.commons.compress.compressors.pack200.Pack200Utils
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorOutputStream.java:[128,26] package Pack200 does not exist
[ERROR] /dataset/src/main/java/org/apache/commons/compress/compressors/pack200/Pack200CompressorOutputStream.java:[128,38] cannot find symbol
  symbol:   variable Pack200
  location: class org.apache.commons.compress.compressors.pack200.Pack200CompressorOutputStream
[INFO] 12 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
--snip--

All 14 failures look like this:

wtwhite@wtwhite-vuw-vm:~/code/jcompile$ grep -L 'Pack200.* cannot find symbol' `cat last_build_failures.txt`|wc -l
0
wtwhite commented 1 year ago

The last 14 errors involving Pack200 are due to it being removed in JDK 14.

There doesn't seem to be an obvious way to get the functionality back, and I don't think it's worth bothering to try.

wtwhite commented 1 year ago

Coming back to the 42 compilations that fail due to specifying JDK6, just adding -Dmaven.compiler.source=7 -Dmaven.compiler.target=7 fixes the compilation step but they fail later at the animal.sniffer plugin:

wtwhite@wtwhite-vuw-vm:~/code/jcompile/worktrees/openjdk-17_commons-codec-1.11_MANUALLY_BY_TIM$ mvn -Dmaven.compiler.source=7 -Dmaven.compiler.target=7 -Dmaven.repo.local=/home/wtwhite/code/jcompile/.m2 -Drat.skip=true -Dmaven.test.skip=true -Dmaven.javadoc.skip=true -Dcyclonedx.skip=true clean package
[INFO] Scanning for projects...
[WARNING] The project commons-codec:commons-codec:jar:1.11 uses prerequisites which is only intended for maven-plugin projects but not for non maven-plugin projects. For such purposes you should use the maven-enforcer-plugin. See https://maven.apache.org/enforcer/enforcer-rules/requireMavenVersion.html
[INFO] 
[INFO] --------------------< commons-codec:commons-codec >---------------------
[INFO] Building Apache Commons Codec 1.11
--snip--
[INFO] --- maven-compiler-plugin:3.6.0:compile (default-compile) @ commons-codec ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 64 source files to /home/wtwhite/code/jcompile/worktrees/openjdk-17_commons-codec-1.11_MANUALLY_BY_TIM/target/classes
[INFO] 
[INFO] --- maven-bundle-plugin:3.0.1:manifest (bundle-manifest) @ commons-codec ---
[INFO] 
[INFO] --- animal-sniffer-maven-plugin:1.15:check (checkAPIcompatibility) @ commons-codec ---
[INFO] Checking unresolved references to org.codehaus.mojo.signature:java70:1.0
Downloading from central: https://repo.maven.apache.org/maven2/org/codehaus/mojo/signature/java70/1.0/java70-1.0.signature
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.215 s
[INFO] Finished at: 2023-10-19T19:55:04+13:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:animal-sniffer-maven-plugin:1.15:check (checkAPIcompatibility) on project commons-codec: Failed to obtain signature: org.codehaus.mojo.signature:java70:1.0: Could not find artifact org.codehaus.mojo.signature:java70:signature:1.0 in central (https://repo.maven.apache.org/maven2)
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command: 
[ERROR]     mvn install:install-file -DgroupId=org.codehaus.mojo.signature -DartifactId=java70 -Dversion=1.0 -Dpackaging=signature -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there: 
[ERROR]     mvn deploy:deploy-file -DgroupId=org.codehaus.mojo.signature -DartifactId=java70 -Dversion=1.0 -Dpackaging=signature -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] 
[ERROR]   org.codehaus.mojo.signature:java70:signature:1.0
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR]   apache.snapshots (https://repository.apache.org/snapshots, releases=false, snapshots=true),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException