tweag / sparkle

Haskell on Apache Spark.
BSD 3-Clause "New" or "Revised" License
447 stars 30 forks source link

Fix spark-submit failures in MacOS in CI #174

Open facundominguez opened 1 year ago

facundominguez commented 1 year ago

When working on #172, the following error was found

Exception in thread "main" java.lang.UnsatisfiedLinkError: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/clodl-top0: dlopen(/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/clodl-top0, 1): Library not loaded: @loader_path/libHSsparkle-lib-ghc9.0.2.dylib
  Referenced from: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/sparkle-app-4130550189215234367/libHSappsZShelloZShello-hs-ghc9.0.2.dylib
  Reason: image not found
    at java.lang.ClassLoader$NativeLibrary.load(Native Method)
    at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
    at java.lang.Runtime.load0(Runtime.java:782)
    at java.lang.System.load(System.java:1100)
    at io.tweag.sparkle.SparkleBase.loadApplication(SparkleBase.java:63)
    at io.tweag.sparkle.SparkleBase.<clinit>(SparkleBase.java:28)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:816)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
23/01/18 13:33:10 INFO util.ShutdownHookManager: Shutdown hook called
23/01/18 13:33:10 INFO util.ShutdownHookManager: Deleting directory /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/spark-8e2c2e74-029d-4f11-9c9b-99e9e1e719c3

This ticket is about diagnosing and fixing such failure.