search

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics
https://arrow.apache.org/
Apache License 2.0
14.68k stars 3.56k forks source link

[CI][Packaging] Java jars job is currently failing on arm64 #41577

Closed raulcd closed 6 months ago

raulcd commented 6 months ago

Describe the bug, including details regarding any error messages, version, and platform.

The java-jars jobs is failing with:

 [INFO] <<< source:3.3.0:jar (default-cli) < generate-sources @ module-info-compiler-maven-plugin <<<
[INFO] 
[INFO] 
[INFO] --- source:3.3.0:jar (default-cli) @ module-info-compiler-maven-plugin ---
Error:  We have duplicated artifacts attached.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Arrow Java Root POM 17.0.0-SNAPSHOT:
[INFO] 
[INFO] Arrow Maven Plugins ................................ SUCCESS [ 35.403 s]
[INFO] Module Info Compiler Maven Plugin .................. FAILURE [ 17.883 s]
[INFO] Arrow Bill of Materials ............................ SKIPPED
[INFO] Apache Arrow Java Root POM ......................... SKIPPED
[INFO] Arrow Format ....................................... SKIPPED
[INFO] Arrow Memory ....................................... SKIPPED
[INFO] Arrow Memory - Core ................................ SKIPPED
[INFO] Arrow Memory - Unsafe .............................. SKIPPED
[INFO] Arrow Memory - Netty Buffer ........................ SKIPPED
[INFO] Arrow Memory - Netty ............................... SKIPPED
[INFO] Arrow Vectors ...................................... SKIPPED
[INFO] Arrow Compression .................................. SKIPPED
[INFO] Arrow Tools ........................................ SKIPPED
[INFO] Arrow JDBC Adapter ................................. SKIPPED
[INFO] Arrow Flight ....................................... SKIPPED
[INFO] Arrow Flight Core .................................. SKIPPED
[INFO] Arrow Flight SQL ................................... SKIPPED
[INFO] Arrow Flight SQL JDBC Driver Core .................. SKIPPED
[INFO] Arrow Flight SQL JDBC Driver ....................... SKIPPED
[INFO] Arrow Flight Integration Tests ..................... SKIPPED
[INFO] Arrow AVRO Adapter ................................. SKIPPED
[INFO] Arrow Algorithms ................................... SKIPPED
[INFO] Arrow Performance Benchmarks ....................... SKIPPED
[INFO] Arrow Java C Data Interface ........................ SKIPPED
[INFO] Arrow Orc Adapter .................................. SKIPPED
[INFO] Arrow Gandiva ...................................... SKIPPED
[INFO] Arrow Java Dataset ................................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  53.821 s
[INFO] Finished at: 2024-05-07T09:29:26Z
[INFO] ------------------------------------------------------------------------
[INFO] 66 goals, 65 executed, 1 from cache
Error:  Failed to execute goal org.apache.maven.plugins:maven-source-plugin:3.3.0:jar (default-cli) on project module-info-compiler-maven-plugin: Presumably you have configured maven-source-plugn to execute twice times in your build. You have to configure a classifier for at least on of them. -> [Help 1]
Error:  
Error:  To see the full stack trace of the errors, re-run Maven with the -e switch.
Error:  Re-run Maven using the -X switch to enable full debug logging.
Error:  
Error:  For more information about the errors and possible solutions, please read the following articles:
Error:  [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Error:  
Error:  After correcting the problems, you can resume the build with the command
Error:    mvn <args> -rf :module-info-compiler-maven-plugin
Error: Process completed with exit code 1.

This seems to be failing since macos-latest moved to arm64.

Component(s)

Continuous Integration, Packaging

raulcd commented 6 months ago

@danepitkin @vibhatha @lidavidm any idea what is the issue on the java-jars? This is currently blocking 16.1.0, if someone can take a look.

lidavidm commented 6 months ago

https://github.com/apache/arrow-java/issues/92

lidavidm commented 6 months ago

Though we didn't cherry-pick the suspected commit so something else is going on?

raulcd commented 6 months ago

ok, let me cherry-pick: https://github.com/apache/arrow/commit/2fd3fcc6e9c2bbd4b105f14306f0ea8bd7d59cec manually and validate

lidavidm commented 6 months ago

I thought that commit didn't fix the issue?

lidavidm commented 6 months ago

I'm saying that linked issue suspects https://github.com/apache/arrow/commit/9090e679da91e0544171c2da9f2b9ce8ba23d389 as the cause. But that's not in 16.1.0, so there's some other cause.

raulcd commented 6 months ago

The release was failing on a slightly different point which might be solved by only adding the other commit I mentioned. I assumed (probably wrongly) that this would also be required for the release. I am testing at the moment, see: https://github.com/apache/arrow/pull/41437#issuecomment-2100012769 I will update if the job succeeds for the release.

vibhatha commented 6 months ago

Yes that commit didn't fix it as far as I understand. I got the same issue in another PR today.

lidavidm commented 6 months ago

@vibhatha can you give this a test? https://stackoverflow.com/questions/76305897/maven-build-fails-after-upgrading-to-maven-source-plugin-from-3-2-1-to-3-3-0

vibhatha commented 6 months ago

Sure @lidavidm, I can.

lidavidm commented 6 months ago

It sounds like either: we can downgrade maven-source-plugin, or we can see if java_full_build.sh is specifying any duplicate phases

vibhatha commented 6 months ago

I looked into the solution, I will probably start testing the ideas within the hour.

raulcd commented 6 months ago

Might also fix: https://github.com/apache/arrow/issues/41571

lidavidm commented 6 months ago

I don't think that's related, I would suspect bad usage of modules on our side or theirs

vibhatha commented 6 months ago

I am testing a few ideas here: https://github.com/apache/arrow/pull/41586 I will update if I get a favorable outcome.

raulcd commented 6 months ago

Adding the following to the maint-16.1.0 branch:

$ git log -p b28633cfe5263ac977598f058b53d76f95d3d1fc
commit b28633cfe5263ac977598f058b53d76f95d3d1fc
Author: Raúl Cumplido <raulcumplido@gmail.com>
Date:   Wed May 8 10:15:17 2024 +0200

    MINOR: [Release] Manually commit fix to avoid cherry-pick conflict for changing runner on java-jars

diff --git a/dev/tasks/java-jars/github.yml b/dev/tasks/java-jars/github.yml
index 0437ee7..eb9478e 100644
--- a/dev/tasks/java-jars/github.yml
+++ b/dev/tasks/java-jars/github.yml
@@ -80,7 +80,7 @@ jobs:
       fail-fast: false
       matrix:
         platform:
-          - { runs_on: ["macos-latest"], arch: "x86_64"}
+          - { runs_on: ["macos-13"], arch: "x86_64"}
           - { runs_on: ["macos-14"], arch: "aarch_64" }
     env:
       MACOSX_DEPLOYMENT_TARGET: "10.15"

commit ac8af222dd2e5800e2c6cbf45a5bdbb2309deed4
Author: Raúl Cumplido <raulcumplido@gmail.com>

fixes the initial issue but fails somewhere else:

Error:  Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.119 s <<< FAILURE! -- in org.apache.arrow.dataset.jni.TestReservationListener
Error:  org.apache.arrow.dataset.jni.TestReservationListener.testDirectReservationListener -- Time elapsed: 0.017 s <<< ERROR!
java.lang.ExceptionInInitializerError
    at org.apache.arrow.dataset/org.apache.arrow.dataset.jni.TestReservationListener.testDirectReservationListener(TestReservationListener.java:46)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.base/java.lang.reflect.Method.invoke(Method.java:580)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
    at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
    at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
    at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
    at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
    at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
    at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:56)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.lambda$execute$1(JUnitPlatformProvider.java:191)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:186)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:148)
    at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:118)
    at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
    at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
    at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
    at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
Caused by: java.lang.RuntimeException: Failed to initialize DirectReservationListener. When starting Java you must include `--add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED` (See https://arrow.apache.org/docs/java/install.html)
    at org.apache.arrow.dataset/org.apache.arrow.dataset.jni.DirectReservationListener.<init>(DirectReservationListener.java:43)
    at org.apache.arrow.dataset/org.apache.arrow.dataset.jni.DirectReservationListener.<clinit>(DirectReservationListener.java:52)
    ... 47 more
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make static void java.nio.Bits.reserveMemory(long,long) accessible: module java.base does not "opens java.nio" to module org.apache.arrow.dataset
    at java.base/java.lang.reflect.AccessibleObject.throwInaccessibleObjectException(AccessibleObject.java:391)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:367)
    at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:315)
    at java.base/java.lang.reflect.Method.checkCanSetAccessible(Method.java:203)
    at java.base/java.lang.reflect.Method.setAccessible(Method.java:197)
    at org.apache.arrow.dataset/org.apache.arrow.dataset.jni.DirectReservationListener.<init>(DirectReservationListener.java:39)
    ... 48 more

[INFO] 
[INFO] Results:
[INFO] 
Error:  Errors: 
Error:    TestReservationListener.testDirectReservationListener:46 ExceptionInInitializer
[INFO] 
Error:  Tests run: 37, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Arrow Java Root POM 17.0.0-SNAPSHOT:
[INFO] 
[INFO] Arrow Maven Plugins ................................ SUCCESS [ 28.471 s]
[INFO] Module Info Compiler Maven Plugin .................. SUCCESS [ 19.556 s]
[INFO] Arrow Bill of Materials ............................ SUCCESS [  4.385 s]
[INFO] Apache Arrow Java Root POM ......................... SUCCESS [ 24.951 s]
[INFO] Arrow Format ....................................... SUCCESS [ 14.401 s]
[INFO] Arrow Memory ....................................... SUCCESS [  3.489 s]
[INFO] Arrow Memory - Core ................................ SUCCESS [ 30.969 s]
[INFO] Arrow Memory - Unsafe .............................. SUCCESS [ 11.935 s]
[INFO] Arrow Memory - Netty Buffer ........................ SUCCESS [ 12.603 s]
[INFO] Arrow Memory - Netty ............................... SUCCESS [ 12.214 s]
[INFO] Arrow Vectors ...................................... SUCCESS [01:52 min]
[INFO] Arrow Compression .................................. SUCCESS [ 14.812 s]
[INFO] Arrow Tools ........................................ SUCCESS [ 17.821 s]
[INFO] Arrow JDBC Adapter ................................. SUCCESS [ 20.325 s]
[INFO] Arrow Flight ....................................... SUCCESS [  4.016 s]
[INFO] Arrow Flight Core .................................. SUCCESS [01:15 min]
[INFO] Arrow Flight SQL ................................... SUCCESS [ 24.863 s]
[INFO] Arrow Flight SQL JDBC Driver Core .................. SUCCESS [02:03 min]
[INFO] Arrow Flight SQL JDBC Driver ....................... SUCCESS [ 39.296 s]
[INFO] Arrow Flight Integration Tests ..................... SUCCESS [ 23.352 s]
[INFO] Arrow AVRO Adapter ................................. SUCCESS [ 16.407 s]
[INFO] Arrow Algorithms ................................... SUCCESS [ 37.601 s]
[INFO] Arrow Performance Benchmarks ....................... SUCCESS [ 12.898 s]
[INFO] Arrow Java C Data Interface ........................ SUCCESS [ 15.981 s]
[INFO] Arrow Orc Adapter .................................. SUCCESS [ 27.511 s]
[INFO] Arrow Gandiva ...................................... SUCCESS [ 54.642 s]
[INFO] Arrow Java Dataset ................................. FAILURE [ 15.656 s]
[INFO] ------------------------------------------------------------------------

I've retried 3 times to see if it was a fluke but seems to be pretty consistent: https://github.com/ursacomputing/crossbow/actions/runs/8998563168/job/24722115895

raulcd commented 6 months ago

Meaning the issue on main is not reproducible on the 16.1.0 release but there's something else to be fixed.

vibhatha commented 6 months ago

I mean if that is the case, the documentation describes the solution

https://arrow.apache.org/docs/java/install.html#java-compatibility

java --add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED -jar ...
env _JAVA_OPTIONS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED" java -jar ...

cc @lidavidm

vibhatha commented 6 months ago

https://github.com/apache/arrow/blob/main/java/dataset/pom.xml#L204

<argLine combine.self="override">--add-reads=org.apache.arrow.dataset=com.fasterxml.jackson.databind --add-opens=java.base/java.nio=org.apache.arrow.dataset,org.apache.arrow.memory.core,ALL-UNNAMED</argLine>

Could this solve?

vibhatha commented 6 months ago

It sounds like either: we can downgrade maven-source-plugin, or we can see if java_full_build.sh is specifying any duplicate phases

This doesn't seem to have duplicate phases, right?

# build the entire project
mvn clean \
    install \
    assembly:single \
    source:jar \
    javadoc:jar \
    -Papache-release \
    -Parrow-c-data \
    -Parrow-jni \
    -Darrow.cpp.build.dir=$dist_dir \
    -Darrow.c.jni.dist.dir=$dist_dir \
    -DdescriptorId=source-release
raulcd commented 6 months ago

If I add the two following commits (manually) https://github.com/apache/arrow/pull/41437/commits/b28633cfe5263ac977598f058b53d76f95d3d1fc and https://github.com/apache/arrow/pull/41437/commits/be62f6c5717d960282c70f6150725b306919a672 to maint-16.1.0 the java-jars job on it does succeed.

Do we want to add https://github.com/apache/arrow/pull/41437/commits/be62f6c5717d960282c70f6150725b306919a672 to main? It won't fix the current java-jars issue. CC @kou @lidavidm

raulcd commented 6 months ago

Could this solve?

It does solve indeed. Thanks @vibhatha

vibhatha commented 6 months ago

That's great. I am working on a parallel fix on main. Still I couldn't figure out it. But these findings could help to narrow it down, I guess.

kou commented 6 months ago

Ah, how about using macos-12 not macos-13 for now? I think that we don't need --add-opens=... with macos-12 (the previous macos-latest).

vibhatha commented 6 months ago

@raulcd Is https://github.com/apache/arrow-java/issues/92 a blocker for 16.1.0 release?

lidavidm commented 6 months ago

Sorry for missing this...we can add it to main too. It should be OK.

lidavidm commented 6 months ago

I assume it would fail without that once we fix the current java-jars issue on main anyways. (I believe @jbonofre has been looking at that, it sounds like this has been an issue for multiple Apache projects.)

vibhatha commented 6 months ago

@lidavidm I tested a few ideas here https://github.com/apache/arrow/pull/41586

But the maven-source-plugin is not something we have explicitly added, it must be internally used somewhere in 31 pom update? Only place we use it is in gandiva and it has the proper configuration.

Also I am not sure if we have multiple phases on the build script either. Sorry I am still trying to figure out a solution for this.

lidavidm commented 6 months ago

We can let JB look at it then, I thought it was causing problems for the 16.1.0 branch. I assumed we could just override the parent pom and downgrade the plugin. It would be nice to reproduce it locally and see if we can avoid the duplicate artifact in the first place. Of course, the other solution might be to drop Java 8 and drop the plugin, and hope the rest of the build works.

vibhatha commented 6 months ago

I need to see if we can bypass the parent pom using a particular version of a dependency it is using, like we do for regular dependencies. Not very familiar with this part, but I will keep looking in parallel.

raulcd commented 6 months ago

I am going to close this as the 16.1.0 release has been solved. The issue for main can be tracked on the original: https://github.com/apache/arrow-java/issues/92