search

eclipse-ditto / ditto

Eclipse Ditto™: Digital Twin framework of Eclipse IoT - main repository
https://eclipse.dev/ditto/
Eclipse Public License 2.0
678 stars 222 forks source link

Building Ditto on Raspberry Pi (ARM) #725

Closed marioverhaeg closed 4 years ago

marioverhaeg commented 4 years ago

Hi,

I was hoping to use Ditto for a home automation project and run it on a Raspberry Pi, hoping this would be powerful enough for small environments. While building Ditto I run into an error. It seems like it doesn't build on the ARM environment. Is there something wrong in my environment or is this "as-designed"?

Greetings from a Bosch colleague :-)

Summary: [ERROR] : java.lang.UnsatisfiedLinkError: /tmp/swoval-jni/2522452405523835701-libswoval-files0.so: /tm p/swoval-jni/2522452405523835701-libswoval-files0.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform)

Full trace:

[INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 28:48 min [INFO] Finished at: 2020-07-08T21:16:11+02:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:4.1.0:compile (default) on projec t ditto-model-rql-parser: Execution default of goal net.alchim31.maven:scala-maven-plugin:4.1.0:compil e failed: An API incompatibility was encountered while executing net.alchim31.maven:scala-maven-plugin :4.1.0:compile: java.lang.ExceptionInInitializerError: null [ERROR] ----------------------------------------------------- [ERROR] realm = plugin>net.alchim31.maven:scala-maven-plugin:4.1.0 [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy [ERROR] urls[0] = file:/home/pi/.m2/repository/net/alchim31/maven/scala-maven-plugin/4.1.0/scala-maven -plugin-4.1.0.jar [ERROR] urls[1] = file:/home/pi/.m2/repository/org/apache/maven/maven-builder-support/3.3.9/maven-buil der-support-3.3.9.jar [ERROR] urls[2] = file:/home/pi/.m2/repository/com/google/guava/guava/18.0/guava-18.0.jar [ERROR] urls[3] = file:/home/pi/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.21/plexus-in terpolation-1.21.jar [ERROR] urls[4] = file:/home/pi/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar [ERROR] urls[5] = file:/home/pi/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.2/org.ecli pse.sisu.inject-0.3.2.jar [ERROR] urls[6] = file:/home/pi/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.6/pl exus-component-annotations-1.6.jar [ERROR] urls[7] = file:/home/pi/.m2/repository/org/apache/maven/reporting/maven-reporting-api/3.0/mave n-reporting-api-3.0.jar [ERROR] urls[8] = file:/home/pi/.m2/repository/org/eclipse/aether/aether-util/1.0.2.v20150114/aether-u til-1.0.2.v20150114.jar [ERROR] urls[9] = file:/home/pi/.m2/repository/com/google/inject/guice/4.0/guice-4.0-no_aop.jar [ERROR] urls[10] = file:/home/pi/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar [ERROR] urls[11] = file:/home/pi/.m2/repository/org/sonatype/plexus/plexus-sec-dispatcher/1.3/plexus-s ec-dispatcher-1.3.jar [ERROR] urls[12] = file:/home/pi/.m2/repository/org/sonatype/plexus/plexus-cipher/1.4/plexus-cipher-1. 4.jar [ERROR] urls[13] = file:/home/pi/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4 .jar [ERROR] urls[14] = file:/home/pi/.m2/repository/org/apache/maven/shared/maven-dependency-tree/3.0.1/ma ven-dependency-tree-3.0.1.jar [ERROR] urls[15] = file:/home/pi/.m2/repository/org/apache/commons/commons-exec/1.3/commons-exec-1.3.j ar [ERROR] urls[16] = file:/home/pi/.m2/repository/org/codehaus/plexus/plexus-utils/3.2.0/plexus-utils-3. 2.0.jar [ERROR] urls[17] = file:/home/pi/.m2/repository/org/codehaus/plexus/plexus-archiver/4.1.0/plexus-archi ver-4.1.0.jar [ERROR] urls[18] = file:/home/pi/.m2/repository/org/codehaus/plexus/plexus-io/3.1.1/plexus-io-3.1.1.ja r [ERROR] urls[19] = file:/home/pi/.m2/repository/org/apache/commons/commons-compress/1.18/commons-compr ess-1.18.jar [ERROR] urls[20] = file:/home/pi/.m2/repository/org/iq80/snappy/snappy/0.4/snappy-0.4.jar [ERROR] urls[21] = file:/home/pi/.m2/repository/org/tukaani/xz/1.8/xz-1.8.jar [ERROR] urls[22] = file:/home/pi/.m2/repository/backport-util-concurrent/backport-util-concurrent/3.1/ backport-util-concurrent-3.1.jar [ERROR] urls[23] = file:/home/pi/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar [ERROR] urls[24] = file:/home/pi/.m2/repository/org/apache/maven/maven-archiver/3.4.0/maven-archiver-3 .4.0.jar [ERROR] urls[25] = file:/home/pi/.m2/repository/org/apache/maven/shared/maven-shared-utils/3.2.1/maven -shared-utils-3.2.1.jar [ERROR] urls[26] = file:/home/pi/.m2/repository/commons-io/commons-io/2.5/commons-io-2.5.jar [ERROR] urls[27] = file:/home/pi/.m2/repository/org/apache/maven/doxia/doxia-sink-api/1.9/doxia-sink-a pi-1.9.jar [ERROR] urls[28] = file:/home/pi/.m2/repository/org/apache/maven/doxia/doxia-logging-api/1.9/doxia-log ging-api-1.9.jar [ERROR] urls[29] = file:/home/pi/.m2/repository/org/apache/maven/shared/maven-invoker/3.0.1/maven-invo ker-3.0.1.jar [ERROR] urls[30] = file:/home/pi/.m2/repository/org/scala-sbt/zinc_2.12/1.3.0-M7/zinc_2.12-1.3.0-M7.ja r [ERROR] urls[31] = file:/home/pi/.m2/repository/org/scala-lang/scala-library/2.12.8/scala-library-2.12 .8.jar [ERROR] urls[32] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-core_2.12/1.3.0-M7/zinc-core_2.12-1 .3.0-M7.jar [ERROR] urls[33] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-apiinfo2.12/1.3.0-M7/zinc-apiinfo 2.12-1.3.0-M7.jar [ERROR] urls[34] = file:/home/pi/.m2/repository/org/scala-sbt/compiler-bridge_2.12/1.3.0-M7/compiler-b ridge_2.12-1.3.0-M7.jar [ERROR] urls[35] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-classpath_2.12/1.3.0-M7/zinc-classp ath_2.12-1.3.0-M7.jar [ERROR] urls[36] = file:/home/pi/.m2/repository/org/scala-lang/scala-compiler/2.12.8/scala-compiler-2. 12.8.jar [ERROR] urls[37] = file:/home/pi/.m2/repository/org/scala-sbt/compiler-interface/1.3.0-M7/compiler-int erface-1.3.0-M7.jar [ERROR] urls[38] = file:/home/pi/.m2/repository/org/scala-sbt/util-interface/1.3.0-M6/util-interface-1 .3.0-M6.jar [ERROR] urls[39] = file:/home/pi/.m2/repository/com/trueaccord/scalapb/scalapb-runtime_2.12/0.6.0/scal apb-runtime_2.12-0.6.0.jar [ERROR] urls[40] = file:/home/pi/.m2/repository/com/trueaccord/lenses/lenses_2.12/0.4.12/lenses_2.12-0 .4.12.jar [ERROR] urls[41] = file:/home/pi/.m2/repository/com/lihaoyi/fastparse_2.12/0.4.2/fastparse_2.12-0.4.2. jar [ERROR] urls[42] = file:/home/pi/.m2/repository/com/lihaoyi/fastparse-utils_2.12/0.4.2/fastparse-utils _2.12-0.4.2.jar [ERROR] urls[43] = file:/home/pi/.m2/repository/com/lihaoyi/sourcecode_2.12/0.1.3/sourcecode_2.12-0.1. 3.jar [ERROR] urls[44] = file:/home/pi/.m2/repository/org/scala-sbt/io_2.12/1.3.0-M10/io_2.12-1.3.0-M10.jar [ERROR] urls[45] = file:/home/pi/.m2/repository/com/swoval/file-tree-views/2.1.1/file-tree-views-2.1.1 .jar [ERROR] urls[46] = file:/home/pi/.m2/repository/net/java/dev/jna/jna/4.5.0/jna-4.5.0.jar [ERROR] urls[47] = file:/home/pi/.m2/repository/net/java/dev/jna/jna-platform/4.5.0/jna-platform-4.5.0 .jar [ERROR] urls[48] = file:/home/pi/.m2/repository/org/scala-sbt/util-logging2.12/1.3.0-M6/util-logging 2.12-1.3.0-M6.jar [ERROR] urls[49] = file:/home/pi/.m2/repository/com/eed3si9n/sjson-new-core2.12/0.8.2/sjson-new-core 2.12-0.8.2.jar [ERROR] urls[50] = file:/home/pi/.m2/repository/jline/jline/2.14.6/jline-2.14.6.jar [ERROR] urls[51] = file:/home/pi/.m2/repository/org/apache/logging/log4j/log4j-api/2.11.2/log4j-api-2. 11.2.jar [ERROR] urls[52] = file:/home/pi/.m2/repository/org/apache/logging/log4j/log4j-core/2.11.2/log4j-core- 2.11.2.jar [ERROR] urls[53] = file:/home/pi/.m2/repository/com/lmax/disruptor/3.4.2/disruptor-3.4.2.jar [ERROR] urls[54] = file:/home/pi/.m2/repository/com/eed3si9n/sjson-new-scalajson_2.12/0.8.2/sjson-new- scalajson_2.12-0.8.2.jar [ERROR] urls[55] = file:/home/pi/.m2/repository/com/eed3si9n/shaded-scalajson_2.12/1.0.0-M4/shaded-sca lajson_2.12-1.0.0-M4.jar [ERROR] urls[56] = file:/home/pi/.m2/repository/org/spire-math/jawn-parser_2.12/0.10.4/jawn-parser_2.1 2-0.10.4.jar [ERROR] urls[57] = file:/home/pi/.m2/repository/org/scala-lang/scala-reflect/2.12.8/scala-reflect-2.12 .8.jar [ERROR] urls[58] = file:/home/pi/.m2/repository/org/scala-sbt/util-relation_2.12/1.3.0-M6/util-relatio n_2.12-1.3.0-M6.jar [ERROR] urls[59] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-persist2.12/1.3.0-M7/zinc-persist 2.12-1.3.0-M7.jar [ERROR] urls[60] = file:/home/pi/.m2/repository/org/scala-sbt/sbinary_2.12/0.4.4/sbinary_2.12-0.4.4.ja r [ERROR] urls[61] = file:/home/pi/.m2/repository/org/scala-lang/modules/scala-xml2.12/1.0.5/scala-xml 2.12-1.0.5.jar [ERROR] urls[62] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-compile-core_2.12/1.3.0-M7/zinc-com pile-core_2.12-1.3.0-M7.jar [ERROR] urls[63] = file:/home/pi/.m2/repository/org/scala-sbt/launcher-interface/1.0.0/launcher-interf ace-1.0.0.jar [ERROR] urls[64] = file:/home/pi/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.12/1 .1.2/scala-parser-combinators_2.12-1.1.2.jar [ERROR] urls[65] = file:/home/pi/.m2/repository/org/scala-sbt/util-control2.12/1.3.0-M6/util-control 2.12-1.3.0-M6.jar [ERROR] urls[66] = file:/home/pi/.m2/repository/org/scala-sbt/zinc-classfile_2.12/1.3.0-M7/zinc-classf ile_2.12-1.3.0-M7.jar [ERROR] urls[67] = file:/home/pi/.m2/repository/com/google/protobuf/protobuf-java/3.7.0/protobuf-java- 3.7.0.jar [ERROR] urls[68] = file:/home/pi/.m2/repository/org/scala-lang/modules/scala-java8-compat_2.12/0.9.0/s cala-java8-compat_2.12-0.9.0.jar [ERROR] Number of foreign imports: 1 [ERROR] import: Entry[import from realm ClassRealm[project>org.eclipse.ditto:ditto-json:0-SNAPSHOT, p arent: ClassRealm[maven.api, parent: null]]] [ERROR] [ERROR] ----------------------------------------------------- [ERROR] : java.lang.UnsatisfiedLinkError: /tmp/swoval-jni/2522452405523835701-libswoval-files0.so: /tm p/swoval-jni/2522452405523835701-libswoval-files0.so: cannot open shared object file: No such file or directory (Possible cause: can't load IA 32-bit .so on a ARM-bit platform) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articl es: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :ditto-model-rql-parser

thjaeckle commented 4 years ago

Hi @marioverhaeg

We never tried before building for ARM, so I guess things will go wrong compiling Ditto. But I would love to see that working as well.

Which JDK did you use for building Ditto? You would need the JDK 11 for ARM which is e.g. provided by Adoptopenjdk: https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.7%2B10/OpenJDK11U-jdk_arm_linux_hotspot_11.0.7_10.tar.gz

The error you highlighted Possible cause: can't load IA 32-bit .so on a ARM-bit platform from "swoval-jni" (included in swoval file-tree-views) sounds like the "swoval-jni" only included shared objects for x64 or x86 architectures. I would guess that this "swoval-jni" is a dependency used in "SBT" (the build tool for Scala) which is invoked by the scala-maven-plugin, that's why this also appears in the failure stack.

I think that it should be able to compile Scala code with SBT on ARM architecture (I didn't find anything which states the opposite).

What you could try is using the latest "scala-maven-plugin" (4.4.0) which uses a more up-to-date SBT zinc comiler 1.3.5.

Ditto uses scala-maven-plugin 4.1.0.

If this doesn't help maybe asking a question either in https://github.com/davidB/scala-maven-plugin or in https://github.com/sbt/sbt could help. Then I'm also out of ideas ;)

ffendt commented 4 years ago

Hi @marioverhaeg,

how about just downloading the jar artifacts from maven central and running them?

thjaeckle commented 4 years ago

@ffendt has made a very good point :) The built .jars should also run with an ARM Java. The "starter" jars are executable: see here

marioverhaeg commented 4 years ago

[Let me rephrase my question] That's a good idea. I've downloaded the jars and got them up and running. However, the configuration is a bit unclear to me. I've looked at: https://www.eclipse.org/ditto/architecture-overview.html Your suggestion basically means that I need to setup MongoDB and NGINX next to Ditto. Is there any guidance on the configuration of this set-up?

marioverhaeg commented 4 years ago

Regarding the build process: I've updated to scala-maven-plugin to 4.4.0 but it's stuck again:

`[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.292 s <<< FAILURE! - in org.eclipse.ditto.services.utils.persistence.mongo.indices.IndexInitializerIT [ERROR] org.eclipse.ditto.services.utils.persistence.mongo.indices.IndexInitializerIT Time elapsed: 4.243 s <<< ERROR! java.lang.NullPointerException at org.eclipse.ditto.services.utils.persistence.mongo.indices.IndexInitializerIT.startMongoResource(IndexInitializerIT.java:102)

[INFO] Running org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoReadJournalIT [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.064 s <<< FAILURE! - in org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoReadJournalIT [ERROR] org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoReadJournalIT Time elapsed: 0.015 s <<< ERROR! java.lang.NullPointerException at org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoReadJournalIT.startMongoResource(MongoReadJournalIT.java:60)

[INFO] Running org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoTimestampPersistenceIT [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.042 s <<< FAILURE! - in org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoTimestampPersistenceIT [ERROR] org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoTimestampPersistenceIT Time elapsed: 0.001 s <<< ERROR! java.lang.NullPointerException at org.eclipse.ditto.services.utils.persistence.mongo.streaming.MongoTimestampPersistenceIT.startMongoResource(MongoTimestampPersistenceIT.java:61)

[INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] IndexInitializerIT.startMongoResource:102 » NullPointer [ERROR] MongoReadJournalIT.startMongoResource:60 » NullPointer [ERROR] MongoTimestampPersistenceIT.startMongoResource:61 » NullPointer `

This seems to be in the ditto-services-utils-persistence. I've installed MongoDB to see if this would be some sort of prerequisite but that didn't help.

thjaeckle commented 4 years ago

That's a good idea. I've downloaded the jars and got them up and running. However, the configuration is a bit unclear to me. I've looked at: https://www.eclipse.org/ditto/architecture-overview.html Your suggestion basically means that I need to setup MongoDB and NGINX next to Ditto. Is there any guidance on the configuration of this set-up?

Yes, MongoDB is a hard requirement for Ditto. Nginx acts as an optional reverse proxy in front of Ditto - you need it for example, when you want to authenticate users via Basic auth. If you don't need user authentication (as you are running it locally on a Raspberry this might not be needed), you can also use the pre-authentication header x-ditto-pre-authenticated and add this to your HTTP requests.

Regards the compiling of Ditto on ARM: This should really not be required. If you would manage to get the build running, you would create the "starter" .jar files which I linked above. You would still need to run them.

In the docker-compose file you can see with which parameters the MongoDB and the Ditto services are started:

Then, the Ditto services should come up and form a cluster.

ffendt commented 4 years ago

Hi @marioverhaeg,

Regarding your first question: you can find a docker-compose setup with MongoDB and NGINX in deployment/docker.

Regarding the build process: There should be no need to install MongoDB yourself. If I remember correctly, we're using flapdoodle for unit testing with MongoDB. It should download a binary of MongoDB itself and run it in the unit tests. There should be some logging output what version it tries to download and start. Can you check if that version would be working on your Raspi?

marioverhaeg commented 4 years ago

Thank you for the comments. I'm getting close now and used the starter jars combined with parts of the docker-compose file to get NGINX and MongoDB up and running. My RPI4 only has 2GB of memory and I'm running into some performance issues. I might need to order an 8GB RPI to get all of this up and running on one system.

marioverhaeg commented 4 years ago

I gave in... I ordered a small Intel NUC and am running Ditto now on an Intel platform. I needed a bit more processing power anyway :-). The RPIs will serve as "edge" nodes.