search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.37k stars 3.49k forks source link

[Java] Implement arrow-memory-ffm #37739

Open danepitkin opened 1 year ago

danepitkin commented 1 year ago

Describe the enhancement requested

Add a memory module that uses the java.lang.foreign module (currently in preview, APIs updated in Java 21).

When using Java 9 or later with the current implementations, some JDK internals must be exposed by adding --add-opens=java.base/java.nio=ALL-UNNAMED to the java command[1]. Here's an article that explains the general Java issue here[2].

Can using java.lang.foreign remove this dependency so users do not have to run with custom options ?

[1]https://arrow.apache.org/docs/java/install.html#java-compatibility [2]https://community.snowflake.com/s/article/JDBC-Driver-Compatibility-Issue-With-JDK-16-and-Later

Component(s)

Java

lidavidm commented 1 year ago

Not until Java 21+ (or really Java 24, when the new memory APIs will presumably be stabilized)

lidavidm commented 1 year ago

Also, this is a runtime flag, not a build flag

danepitkin commented 1 year ago

Ah I see. Updated to reference runtime now. Shall I close this given this won't happen any time soon?

lidavidm commented 1 year ago

The interesting things to me would be (1) implementing an arrow-memory-ffm that uses the experimental API so we can figure out how well it works and (2) refactoring BufferAllocator and other core APIs to be in line with the new upstream APIs so that when Java 24 comes, we're ready

lidavidm commented 1 year ago

Because if something is missing for us it's kinda too late to communicate any feedback upstream