How to build for the aarch64?

[huangtianhua]

When I execute 'mvn test' for apache/spark project, I receive an error like: [ERROR] copyIndexDescendingWithStart(org.apache.spark.util.kvstore.LevelDBIteratorSuite) Time elapsed: 0.2 s <<< ERROR! java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 in java.library.path, no leveldbjni-1.8 in java.library.path, no leveldbjni in java.library.path, /usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8: /usr/local/src/spark/common/kvstore/target/tmp/libleveldbjni-64-1-610267671268036503.8: cannot open shared object file: No such file or directory (Possible cause: can't load AMD 64-bit .so on a AARCH64-bit platform)] at org.apache.spark.util.kvstore.LevelDBIteratorSuite.createStore(LevelDBIteratorSuite.java:44)

and the detail in http://paste.openstack.org/show/752063/, seems there is no native package for aarch64 in leveldbjni-1.8(all) .jar, I found aarch64 is supported after pr #82, but it was not in the 1.8 release, right? And I can't find any update or release after that. How I can fix this, or how to build it for aarch64? Ask for help, thank you !

[ZhengZhenyu]

@chirino @johnmark @JunHe77

[JunHe77]

@ZhengZhenyu @huangtianhua Pls refer this script: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/do-component-build#L46-L104

[huangtianhua]

@JunHe77 I have already deploy the leveldbjni-all-1.8.jar which support aarch64, see https://search.maven.org/search?q=g:org.openlabtesting.leveldbjni :) I based on the master. The details you can find https://github.com/theopenlab/leveldbjni/commit/aa6b14a50a9acb2681102975107784f052f1ceeb

[ZhengZhenyu]

@JunHe77 Hi Jun, thanks for the info and your previous helps. We are trying to making bigdata ecosystem more compatible with ARM architecture natively from upstream. I know that you and your team are really experienced in this area, I wonder if we can cooperate ion this goal. We have sent out maillists and reported issues in projects like Hadoop, Spark, Flink etc, is it possible for you guys to show some supports in those mails and issues? :)

[ZhengZhenyu]

@JunHe77 FYI http://hadoop-common.472056.n3.nabble.com/DISCUSS-ARM-aarch64-support-for-Hadoop-td4190435.html

[KaloyanTanev]

Did somebody try running @huangtianhua's solution on Apple M1 chip? It should be the same architecture (64-bit ARM), but still got the same error: Could not load any of the factory classes: org.openlabtesting.leveldbjni.JniDBFactory, org.iq80.leveldb.impl.Iq80DBFactory.

I am using sbt and add as a dependency "org.openlabtesting.leveldbjni" % "leveldbjni-linux64-aarch64" % "1.8". Then in my class LevelDBFactory I have: private val nativeFactory = "org.openlabtesting.leveldbjni.JniDBFactory" private val javaFactory = "org.iq80.leveldb.impl.Iq80DBFactory".

[csaltos]

Did somebody try running @huangtianhua's solution on Apple M1 chip? It should be the same architecture (64-bit ARM), but still got the same error: Could not load any of the factory classes: org.openlabtesting.leveldbjni.JniDBFactory, org.iq80.leveldb.impl.Iq80DBFactory.

Sadly they are a different target, while they are based on the same architecture (ARM64) they are different operating systems ... is like having a computer running Windows on an Intel chip and a Macintosh running on an Intel chip ... it's the same chip but totally different operating systems where a Windows executable will not work on a Macintosh machine or a Linux on a Mac (at least directly in native mode)

[csaltos]

@JunHe77 I have already deploy the leveldbjni-all-1.8.jar which support aarch64, see https://search.maven.org/search?q=g:org.openlabtesting.leveldbjni :) I based on the master. The details you can find theopenlab@aa6b14a

Thank you very much !! ... You are a big data hero !! ;) :)