search

microsoft / SynapseML

Simple and Distributed Machine Learning
http://aka.ms/spark
MIT License
5.06k stars 830 forks source link

Exception in loading lightgbm library #1507

Closed Quentin-Song closed 2 years ago

Quentin-Song commented 2 years ago

User reported the following stack trace:

/Users/songqingqiang/soft/anaconda3/envs/torch_38/bin/python /Users/songqingqiang/workProject/dev/xiaodan/er.py
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/jars/spark-unsafe_2.12-3.2.1.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
https://mmlspark.azureedge.net/maven added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/songqingqiang/.ivy2/cache
The jars for the packages stored in: /Users/songqingqiang/.ivy2/jars
com.microsoft.azure#synapseml_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-ccfe1aa8-3f2e-4ecd-a96c-9a956a204e5c;1.0
    confs: [default]
    found com.microsoft.azure#synapseml_2.12;0.9.5 in central
    found com.microsoft.azure#synapseml-core_2.12;0.9.5 in central
    found org.scalactic#scalactic_2.12;3.0.5 in central
    found org.scala-lang#scala-reflect;2.12.4 in central
    found io.spray#spray-json_2.12;1.3.2 in central
    found com.jcraft#jsch;0.1.54 in central
    found org.apache.httpcomponents#httpclient;4.5.6 in central
    found org.apache.httpcomponents#httpcore;4.4.10 in central
    found commons-logging#commons-logging;1.2 in central
    found commons-codec#commons-codec;1.10 in central
    found org.apache.httpcomponents#httpmime;4.5.6 in central
    found com.linkedin.isolation-forest#isolation-forest_3.2.0_2.12;2.0.8 in central
    found com.chuusai#shapeless_2.12;2.3.2 in central
    found org.typelevel#macro-compat_2.12;1.1.1 in central
    found org.apache.spark#spark-avro_2.12;3.2.0 in central
    found org.tukaani#xz;1.8 in central
    found org.spark-project.spark#unused;1.0.0 in central
    found org.testng#testng;6.8.8 in central
    found org.beanshell#bsh;2.0b4 in central
    found com.beust#jcommander;1.27 in central
    found com.microsoft.azure#synapseml-deep-learning_2.12;0.9.5 in central
    found com.microsoft.azure#synapseml-opencv_2.12;0.9.5 in central
    found org.openpnp#opencv;3.2.0-1 in central
    found com.microsoft.cntk#cntk;2.4 in central
    found com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 in central
    found com.microsoft.azure#synapseml-cognitive_2.12;0.9.5 in central
    found com.microsoft.cognitiveservices.speech#client-jar-sdk;1.14.0 in central
    found com.azure#azure-storage-blob;12.14.2 in central
    found com.azure#azure-core;1.22.0 in central
    found com.fasterxml.jackson.core#jackson-annotations;2.12.5 in central
    found com.fasterxml.jackson.core#jackson-core;2.12.5 in central
    found com.fasterxml.jackson.core#jackson-databind;2.12.5 in central
    found com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.12.5 in central
    found com.fasterxml.jackson.dataformat#jackson-dataformat-xml;2.12.5 in central
    found com.fasterxml.jackson.module#jackson-module-jaxb-annotations;2.12.5 in central
    found jakarta.xml.bind#jakarta.xml.bind-api;2.3.2 in central
    found jakarta.activation#jakarta.activation-api;1.2.1 in central
    found org.codehaus.woodstox#stax2-api;4.2.1 in central
    found com.fasterxml.woodstox#woodstox-core;6.2.4 in central
    found org.slf4j#slf4j-api;1.7.32 in central
    found io.projectreactor#reactor-core;3.4.10 in central
    found org.reactivestreams#reactive-streams;1.0.3 in central
    found io.netty#netty-tcnative-boringssl-static;2.0.43.Final in central
    found com.azure#azure-core-http-netty;1.11.2 in central
    found io.netty#netty-handler;4.1.68.Final in central
    found io.netty#netty-common;4.1.68.Final in central
    found io.netty#netty-resolver;4.1.68.Final in central
    found io.netty#netty-buffer;4.1.68.Final in central
    found io.netty#netty-transport;4.1.68.Final in central
    found io.netty#netty-codec;4.1.68.Final in central
    found io.netty#netty-handler-proxy;4.1.68.Final in central
    found io.netty#netty-codec-socks;4.1.68.Final in central
    found io.netty#netty-codec-http;4.1.68.Final in central
    found io.netty#netty-codec-http2;4.1.68.Final in central
    found io.netty#netty-transport-native-unix-common;4.1.68.Final in central
    found io.netty#netty-transport-native-epoll;4.1.68.Final in central
    found io.netty#netty-transport-native-kqueue;4.1.68.Final in central
    found io.projectreactor.netty#reactor-netty-http;1.0.11 in central
    found io.netty#netty-resolver-dns;4.1.68.Final in central
    found io.netty#netty-codec-dns;4.1.68.Final in central
    found io.netty#netty-resolver-dns-native-macos;4.1.68.Final in central
    found io.projectreactor.netty#reactor-netty-core;1.0.11 in central
    found com.azure#azure-storage-common;12.14.1 in central
    found com.azure#azure-storage-internal-avro;12.1.2 in central
    found com.azure#azure-ai-textanalytics;5.1.4 in central
    found com.microsoft.azure#synapseml-vw_2.12;0.9.5 in central
    found com.github.vowpalwabbit#vw-jni;8.9.1 in central
    found com.microsoft.azure#synapseml-lightgbm_2.12;0.9.5 in central
    found com.microsoft.ml.lightgbm#lightgbmlib;3.2.110 in central
:: resolution report :: resolve 636ms :: artifacts dl 16ms
    :: modules in use:
    com.azure#azure-ai-textanalytics;5.1.4 from central in [default]
    com.azure#azure-core;1.22.0 from central in [default]
    com.azure#azure-core-http-netty;1.11.2 from central in [default]
    com.azure#azure-storage-blob;12.14.2 from central in [default]
    com.azure#azure-storage-common;12.14.1 from central in [default]
    com.azure#azure-storage-internal-avro;12.1.2 from central in [default]
    com.beust#jcommander;1.27 from central in [default]
    com.chuusai#shapeless_2.12;2.3.2 from central in [default]
    com.fasterxml.jackson.core#jackson-annotations;2.12.5 from central in [default]
    com.fasterxml.jackson.core#jackson-core;2.12.5 from central in [default]
    com.fasterxml.jackson.core#jackson-databind;2.12.5 from central in [default]
    com.fasterxml.jackson.dataformat#jackson-dataformat-xml;2.12.5 from central in [default]
    com.fasterxml.jackson.datatype#jackson-datatype-jsr310;2.12.5 from central in [default]
    com.fasterxml.jackson.module#jackson-module-jaxb-annotations;2.12.5 from central in [default]
    com.fasterxml.woodstox#woodstox-core;6.2.4 from central in [default]
    com.github.vowpalwabbit#vw-jni;8.9.1 from central in [default]
    com.jcraft#jsch;0.1.54 from central in [default]
    com.linkedin.isolation-forest#isolation-forest_3.2.0_2.12;2.0.8 from central in [default]
    com.microsoft.azure#synapseml-cognitive_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-core_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-deep-learning_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-lightgbm_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-opencv_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml-vw_2.12;0.9.5 from central in [default]
    com.microsoft.azure#synapseml_2.12;0.9.5 from central in [default]
    com.microsoft.cntk#cntk;2.4 from central in [default]
    com.microsoft.cognitiveservices.speech#client-jar-sdk;1.14.0 from central in [default]
    com.microsoft.ml.lightgbm#lightgbmlib;3.2.110 from central in [default]
    com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 from central in [default]
    commons-codec#commons-codec;1.10 from central in [default]
    commons-logging#commons-logging;1.2 from central in [default]
    io.netty#netty-buffer;4.1.68.Final from central in [default]
    io.netty#netty-codec;4.1.68.Final from central in [default]
    io.netty#netty-codec-dns;4.1.68.Final from central in [default]
    io.netty#netty-codec-http;4.1.68.Final from central in [default]
    io.netty#netty-codec-http2;4.1.68.Final from central in [default]
    io.netty#netty-codec-socks;4.1.68.Final from central in [default]
    io.netty#netty-common;4.1.68.Final from central in [default]
    io.netty#netty-handler;4.1.68.Final from central in [default]
    io.netty#netty-handler-proxy;4.1.68.Final from central in [default]
    io.netty#netty-resolver;4.1.68.Final from central in [default]
    io.netty#netty-resolver-dns;4.1.68.Final from central in [default]
    io.netty#netty-resolver-dns-native-macos;4.1.68.Final from central in [default]
    io.netty#netty-tcnative-boringssl-static;2.0.43.Final from central in [default]
    io.netty#netty-transport;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-epoll;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-kqueue;4.1.68.Final from central in [default]
    io.netty#netty-transport-native-unix-common;4.1.68.Final from central in [default]
    io.projectreactor#reactor-core;3.4.10 from central in [default]
    io.projectreactor.netty#reactor-netty-core;1.0.11 from central in [default]
    io.projectreactor.netty#reactor-netty-http;1.0.11 from central in [default]
    io.spray#spray-json_2.12;1.3.2 from central in [default]
    jakarta.activation#jakarta.activation-api;1.2.1 from central in [default]
    jakarta.xml.bind#jakarta.xml.bind-api;2.3.2 from central in [default]
    org.apache.httpcomponents#httpclient;4.5.6 from central in [default]
    org.apache.httpcomponents#httpcore;4.4.10 from central in [default]
    org.apache.httpcomponents#httpmime;4.5.6 from central in [default]
    org.apache.spark#spark-avro_2.12;3.2.0 from central in [default]
    org.beanshell#bsh;2.0b4 from central in [default]
    org.codehaus.woodstox#stax2-api;4.2.1 from central in [default]
    org.openpnp#opencv;3.2.0-1 from central in [default]
    org.reactivestreams#reactive-streams;1.0.3 from central in [default]
    org.scala-lang#scala-reflect;2.12.4 from central in [default]
    org.scalactic#scalactic_2.12;3.0.5 from central in [default]
    org.slf4j#slf4j-api;1.7.32 from central in [default]
    org.spark-project.spark#unused;1.0.0 from central in [default]
    org.testng#testng;6.8.8 from central in [default]
    org.tukaani#xz;1.8 from central in [default]
    org.typelevel#macro-compat_2.12;1.1.1 from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   69  |   0   |   0   |   0   ||   69  |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-ccfe1aa8-3f2e-4ecd-a96c-9a956a204e5c
    confs: [default]
    0 artifacts copied, 69 already retrieved (0kB/10ms)
22/05/10 10:17:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
22/05/10 10:17:39 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
22/05/10 10:17:39 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
22/05/10 10:18:01 ERROR Executor: Exception in task 7.0 in stage 14.0 (TID 52)
java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
22/05/10 10:18:01 WARN TaskSetManager: Lost task 7.0 in stage 14.0 (TID 52) (192.168.1.2 executor driver): java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

22/05/10 10:18:01 ERROR TaskSetManager: Task 7 in stage 14.0 failed 1 times; aborting job
22/05/10 10:18:01 ERROR LightGBMClassifier: {"uid":"LightGBMClassifier_acb57948aea6","className":"class com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier","method":"train","buildVersion":"0.9.5"}
org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 14.0 failed 1 times, most recent failure: Lost task 7.0 in stage 14.0 (TID 52) (192.168.1.2 executor driver): java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2454)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2403)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2402)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2402)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1160)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1160)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1160)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2642)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2584)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2573)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:938)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2214)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2309)
    at org.apache.spark.rdd.RDD.$anonfun$reduce$1(RDD.scala:1120)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
    at org.apache.spark.rdd.RDD.reduce(RDD.scala:1102)
    at org.apache.spark.sql.Dataset.$anonfun$reduce$1(Dataset.scala:1738)
    at org.apache.spark.sql.Dataset.$anonfun$withNewRDDExecutionId$1(Dataset.scala:3695)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3693)
    at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1738)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain(LightGBMBase.scala:490)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain$(LightGBMBase.scala:447)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.innerTrain(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$1(LightGBMBase.scala:63)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:63)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:60)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logVerb(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:49)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:48)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logTrain(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:44)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:43)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:26)
    at org.apache.spark.ml.Predictor.fit(Predictor.scala:151)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more
Traceback (most recent call last):
  File "/Users/songqingqiang/workProject/dev/xiaodan/er.py", line 182, in <module>
    model = pipeline.fit(traindf)
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/ml/pipeline.py", line 114, in _fit
    model = stage.fit(dataset)
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/ml/base.py", line 161, in fit
    return self._fit(dataset)
  File "/private/var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/spark-78fb8214-a30a-47f3-a070-d17d2861d886/userFiles-c6c1b2d2-0334-48e0-847a-2db791536759/com.microsoft.azure_synapseml-lightgbm_2.12-0.9.5.jar/synapse/ml/lightgbm/LightGBMClassifier.py", line 1469, in _fit
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/ml/wrapper.py", line 332, in _fit_java
    return self._java_obj.fit(dataset._jdf)
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/py4j/java_gateway.py", line 1309, in __call__
    return_value = get_return_value(
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/pyspark/sql/utils.py", line 111, in deco
    return f(*a, **kw)
  File "/Users/songqingqiang/soft/anaconda3/envs/torch_38/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value
    raise Py4JJavaError(
py4j.protocol.Py4JJavaError: An error occurred while calling o230.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 14.0 failed 1 times, most recent failure: Lost task 7.0 in stage 14.0 (TID 52) (192.168.1.2 executor driver): java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2454)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2403)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2402)
    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2402)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1160)
    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1160)
    at scala.Option.foreach(Option.scala:407)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1160)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2642)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2584)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2573)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:938)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2214)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2309)
    at org.apache.spark.rdd.RDD.$anonfun$reduce$1(RDD.scala:1120)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
    at org.apache.spark.rdd.RDD.reduce(RDD.scala:1102)
    at org.apache.spark.sql.Dataset.$anonfun$reduce$1(Dataset.scala:1738)
    at org.apache.spark.sql.Dataset.$anonfun$withNewRDDExecutionId$1(Dataset.scala:3695)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
    at org.apache.spark.sql.Dataset.withNewRDDExecutionId(Dataset.scala:3693)
    at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1738)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain(LightGBMBase.scala:490)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.innerTrain$(LightGBMBase.scala:447)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.innerTrain(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$train$1(LightGBMBase.scala:63)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb(BasicLogging.scala:63)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logVerb$(BasicLogging.scala:60)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logVerb(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain(BasicLogging.scala:49)
    at com.microsoft.azure.synapse.ml.logging.BasicLogging.logTrain$(BasicLogging.scala:48)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.logTrain(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train(LightGBMBase.scala:44)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.train$(LightGBMBase.scala:43)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:26)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMClassifier.train(LightGBMClassifier.scala:26)
    at org.apache.spark.ml.Predictor.fit(Predictor.scala:151)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.UnsatisfiedLinkError: Can't load library: /var/folders/nm/gh3n43z51j96ly6bq_xsr65r0000gn/T/mml-natives3029460519400283438/lib_lightgbm.dylib
    at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2633)
    at java.base/java.lang.Runtime.load0(Runtime.java:768)
    at java.base/java.lang.System.load(System.java:1837)
    at com.microsoft.azure.synapse.ml.core.env.NativeLoader.loadLibraryByName(NativeLoader.java:66)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:39)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.trainLightGBM(LightGBMBase.scala:356)
    at com.microsoft.azure.synapse.ml.lightgbm.LightGBMBase.$anonfun$innerTrain$4(LightGBMBase.scala:485)
    at org.apache.spark.sql.execution.MapPartitionsExec.$anonfun$doExecute$3(objects.scala:201)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:898)
    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:898)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.sql.execution.SQLExecutionRDD.compute(SQLExecutionRDD.scala:55)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:373)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:337)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:131)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1462)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    ... 1 more

22/05/10 10:18:01 WARN TaskSetManager: Lost task 2.0 in stage 14.0 (TID 47) (192.168.1.2 executor driver): TaskKilled (Stage cancelled)
22/05/10 10:18:02 WARN TaskSetManager: Lost task 6.0 in stage 14.0 (TID 51) (192.168.1.2 executor driver): TaskKilled (Stage cancelled)
22/05/10 10:18:02 WARN TaskSetManager: Lost task 1.0 in stage 14.0 (TID 46) (192.168.1.2 executor driver): TaskKilled (Stage cancelled)

Process finished with exit code 1

AB#1784670

KeerthiYandaOS commented 2 years ago

@Quentin-Song Can you please share when you are seeing this error and what were you trying to execute? I see a stack trace posted in the description but can you post the additional details to help us better understand the scenario.

ppruthi commented 2 years ago

@Quentin-Song -- can you respond the the request above ?

svotaw commented 2 years ago

There were some bugs for a while in main that resulting in library loading error, so this could be related. Please use version 0.10.1 if this is still an issue. closing this for now.