[Java][JPMS][IDE] Fix developer docs for building Arrow Java JPMS in IntelliJ IDE

[davisusanibar]

Describe the enhancement requested

The process of executing mvn clean install did not encounter any problems when we were trying to build Java modules by command line (i.e. Github CI).

The following error appears if we are trying to run and debug tests using an IDE.

/arrow/java/memory/memory-core/src/main/java/module-info.java:18
java: modules are not supported in -source 8
  (use -source 9 or higher to enable modules)

Component(s)

Java

[davisusanibar]

There may not be an option to continue using IntelliJ JPS build for our Java modules project.

IntelliJ offers the following options for compiling Java projects:

1. Compile using maven commands by "View --> Tool Window --> Maven --> Run maven build" : OK
2. Compile using IntelliJ own JPS compiler by "Build --> Build Project" : NOK ( modules are not supported in -source 8)
3. Compile using terminal "View --> Tool Window --> Terminal --> Run maven build" : OK

Option (1) and (3) work without errors because these options honors maven/pom.xml definitions.

Option (2) uses their own native compiler. IntelliJ IDEA compiles code using a tool called JPS (JetBrains Project System). JPS is designed to speed up compiling by performing incremental and background compilations.

To continue using option (2) we need to "Delegate build and run actions to Maven" using these steps.

[davisusanibar]

Delegating build and run actions to Maven will result in the following features being lost:

• Incremental Compilation: JPS tracks changes in your project as you make modifications. When you compile, instead of recompiling your entire project from scratch, JPS compiles only the files that have changed since the last compile and any dependent files. This saves time by skipping the recompilation of unchanged files.
• Background Compilation: As you make changes to your project, JPS compiles those changes in the background. This means that when you run or debug your project, most of your recent changes have already been compiled and are ready to go.
• Distributed and Parallel Compilation: JPS can leverage multiple CPU cores and distribute the compile workload among them for faster compiling. This can be greatly beneficial in large projects with many modules and dependencies.

[davisusanibar]

HI @jduo, James. Do you have any hacks for building Arrow Java modules within IntelliJ?

[davisusanibar]

Would you be able to suggest any next steps to resolve this issue, @danepitkin / @lidavidm:

1. Would it be necessary to spend more time figuring out a workaround to continue using IntelliJ JPS features for the size and variety of our Java modules? , or
2. This issue can be resolved by adding Delegate build and run actions to Maven to our build document.

@vibhatha

[jduo]

I configure my IntelliJ's build to use maven rather than use its own build system:

https://www.jetbrains.com/help/idea/delegate-build-and-run-actions-to-maven.html#delegate_to_maven

[lidavidm]

That option is far, far slower (at least for me) though.

[lidavidm]

I was able to get part of Arrow building inside IntelliJ when I set the source/target level to 21. But I ran into cases where the module-info declarations were incorrect. If we fix those we might be able to create a Maven profile for IntelliJ specifically that lets it use its builtin build system.

[vibhatha]

I was able to get part of Arrow building inside IntelliJ when I set the source/target level to 21. But I ran into cases where the module-info declarations were incorrect. If we fix those we might be able to create a Maven profile for IntelliJ specifically that lets it use its builtin build system.

I observed the same.

[vibhatha]

I went a few steps ahead and got stuck here

/home/arrowuser/github/fork/arrow/java/memory/memory-unsafe/src/main/java/module-info.java:21:35
java: module not found: org.apache.arrow.memory.core

[llama90]

Hello.

Hello. I am interested in Java in addition to the Arrow code, and I'm wondering if there is a way to set up an environment and perform code using IntelliJ?

From what I can see, there doesn't seem to be a clear solution to this issue yet. How do other people configure their environment to work with Java code?

[lidavidm]

Basically, the IDE intelligence will be semi broken and you can only build on command line (or tell IntelliJ to invoke Maven which is basically the same thing in as far as you can't get incremental builds anymore)

[lidavidm]

This seems sufficient for me so far:

diff --git a/java/memory/memory-core/src/main/java/module-info.java b/java/memory/memory-core/src/main/java/module-info.java
index 52fcb52d0..b6364ee36 100644
--- a/java/memory/memory-core/src/main/java/module-info.java
+++ b/java/memory/memory-core/src/main/java/module-info.java
@@ -25,4 +25,7 @@ module org.apache.arrow.memory.core {
   requires transitive jdk.unsupported;
   requires jsr305;
   requires org.slf4j;
+  requires org.checkerframework.checker.qual;
+  requires org.immutables.value.annotations;
+  requires java.compiler; // for javax.annotation.processing
 }
diff --git a/java/pom.xml b/java/pom.xml
index 7c4683a72..0422e71a1 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -112,8 +112,8 @@ under the License.
     <doclint>none</doclint>
     <additionalparam>-Xdoclint:none</additionalparam>
     <!-- org.apache:apache overrides -->
-    <maven.compiler.source>1.8</maven.compiler.source>
-    <maven.compiler.target>1.8</maven.compiler.target>
+    <maven.compiler.source>17</maven.compiler.source>
+    <maven.compiler.target>17</maven.compiler.target>
     <maven.plugin.tools.version>3.12.0</maven.plugin.tools.version>
     <surefire.version>3.2.5</surefire.version>
     <version.apache-rat-plugin>0.16.1</version.apache-rat-plugin>
@@ -308,8 +308,6 @@ under the License.
           <artifactId>maven-compiler-plugin</artifactId>
           <configuration>
             <fork>true</fork>
-            <excludes>**/module-info.java</excludes>
-            <testExcludes>**/module-info.java</testExcludes>
             <useModulePath>false</useModulePath>
             <annotationProcessorPaths>
               <path>
@@ -456,11 +454,6 @@ under the License.
             </sourceFileExcludes>
           </configuration>
         </plugin>
-        <plugin>
-          <groupId>org.apache.arrow.maven.plugins</groupId>
-          <artifactId>module-info-compiler-maven-plugin</artifactId>
-          <version>${project.version}</version>
-        </plugin>
         <plugin>
           <groupId>com.gradle</groupId>
           <artifactId>develocity-maven-extension</artifactId>

[vibhatha]

So As soon as JDK8 is removed, this should be fine?

[lidavidm]

No, it turns out it fails at runtime

[lidavidm]

java.lang.module.FindException: Module org.checkerframework.checker.qual not found, required by org.apache.arrow.memory.core

[vibhatha]

I got this earlier, can you add this to the modules?

[lidavidm]

It's because IntelliJ doesn't add Checker Framework's JAR to the runtime path for some reason.

[vibhatha]

At run-time, for me, Intellij doesn't load any of the Arrow Jars for some sort of an odd reason.

[lidavidm]

I figured out my issue at least.

This is enough to fix IntelliJ

```diff diff --git a/java/memory/memory-core/src/main/java/module-info.java b/java/memory/memory-core/src/main/java/module-info.java index 52fcb52d0..b6364ee36 100644 --- a/java/memory/memory-core/src/main/java/module-info.java +++ b/java/memory/memory-core/src/main/java/module-info.java @@ -25,4 +25,7 @@ module org.apache.arrow.memory.core { requires transitive jdk.unsupported; requires jsr305; requires org.slf4j; + requires org.checkerframework.checker.qual; + requires org.immutables.value.annotations; + requires java.compiler; // for javax.annotation.processing } diff --git a/java/pom.xml b/java/pom.xml index 7c4683a72..f9d421f1d 100644 --- a/java/pom.xml +++ b/java/pom.xml @@ -112,8 +112,8 @@ under the License. none -Xdoclint:none - 1.8 - 1.8 + 17 + 17 3.12.0 3.2.5 0.16.1 @@ -144,7 +144,6 @@ under the License. org.checkerframework checker-qual ${checker.framework.version} - provided com.google.flatbuffers @@ -308,8 +307,6 @@ under the License. maven-compiler-plugin true - **/module-info.java - **/module-info.java false @@ -456,11 +453,6 @@ under the License. - - org.apache.arrow.maven.plugins - module-info-compiler-maven-plugin - ${project.version} - com.gradle develocity-maven-extension ```