tweag / sparkle

Haskell on Apache Spark.
BSD 3-Clause "New" or "Revised" License
447 stars 30 forks source link

"cannot dynamically load executable" when trying to run the hello-world on NixOS #111

Closed thufschmitt closed 7 years ago

thufschmitt commented 7 years ago

When trying to run the sparkle-example-hello program, I get the following error on NixOS (version 18.03pre118381.4068703502):

Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/sparkle-app-2880870777987236962/hsapp: /tmp/sparkle-app-2880870777987236962/hsapp: cannot dynamically load executable
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
    at java.lang.Runtime.load0(Runtime.java:809)
    at java.lang.System.load(System.java:1086)
    at io.tweag.sparkle.SparkleBase.loadApplication(SparkleBase.java:61)
    at io.tweag.sparkle.SparkleBase.<clinit>(SparkleBase.java:28)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:229)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I get these errors whether I run the jar locally with spark-submit or on an aws emr cluster.

Steps to reproduce

From a fresh clone:

export NIX_PATH=nixpkgs=https://github.com/nixos/nixpkgs/archive/3fe7cddc30.tar.gz
stack --nix build hello
stack --nix exec -- sparkle package sparkle-example-hello
stack --nix exec -- spark-submit --master 'local[1]' sparkle-example-hello.jar
thufschmitt commented 7 years ago

635e43f8a86b8a896c4b9447d0f91cd1b8d1ad55 fixed it, thanks!