LightGBM error java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_void;

[lppsuixn]

start a spark shell

spark-shell --packages Azure:mmlspark:0.12

and code like this

scala> import com.microsoft.ml.spark.LightGBMClassifier
import com.microsoft.ml.spark.LightGBMClassifier
scala> 
scala> val model = new LightGBMClassifier().setLabelCol("dep_delayed_15min").setFeaturesCol("newColumn").setPredictionCol("prediction")
model: com.microsoft.ml.spark.LightGBMClassifier = LightGBMClassifier_4a1b178881ad
scala> 
scala> val lrmodel__EE05D07BB334591CB649FB4EECDF9161 = model.fit(tmpData)
java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_void;
  at java.lang.Class.getDeclaredFields0(Native Method)
  at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
  at java.lang.Class.getDeclaredField(Class.java:2068)
  at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1803)
  at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)
  at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:494)
  at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:482)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)
  at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)
  at java.io.ObjectOutputStream.writeClass(ObjectOutputStream.java:1213)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1120)
  at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
  at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
  at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
  at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
  at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)
  at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
  at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
  at org.apache.spark.SparkContext.clean(SparkContext.scala:2094)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:793)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:792)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:792)
  at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:2547)
  at org.apache.spark.sql.Dataset.rdd(Dataset.scala:2544)
  at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1424)
  at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:49)
  at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:22)
  at org.apache.spark.ml.Predictor.fit(Predictor.scala:96)
  ... 56 elided
Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_void
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 95 more

[mhamilton723]

Thanks for trying this out, @imatiach-msft, are there any natives to explicitly load with this?

[imatiach-msft]

@lppsuixn @mhamilton723 it looks like the native so's are not getting loaded, is this being run on a linux machine?

[lppsuixn]

@imatiach-msft yes.it's ubuntu16.04 .And do you have any suggestion for this?Thanks.

[alekcei1510]

I ended with the same issue when i deploy to my cluster.

[imatiach-msft]

@lppsuixn @alekcei1510 sorry I don't have a repro for this and I'm not sure what could be causing it, other than that the native .so's are not getting loaded - but since you are using a linux machine that should not be an issue

[alekcei1510]

I already know what happens for me, when i export like jar file with the command --jars, the swig pointers doesn´t load the native libreries. But when i load as package it works fine for me.

[YangChaoKiKa]

@imatiach-msft , In my macOS, I was trying to run the test code in the corresponding test directory, https://github.com/Azure/mmlspark/blob/master/src/lightgbm/src/test/scala/VerifyLightGBMClassifier.scala#L24 . I got the exception as follows, I have already read https://github.com/Microsoft/LightGBM/issues/1326 and some others, Still No luck, please help, or tell me how to contribute to this issue, thanks a lot scala> val model = lgbm.setLabelCol(labelColumnName).setFeaturesCol(featuresColumn).setRawPredictionCol(rawPredCol).setNumLeaves(5). | setNumIterations(10).fit(featurizer.transform(data)) 18/05/06 22:38:58 ERROR Executor: Exception in task 0.0 in stage 3.0 (TID 3) java.lang.UnsatisfiedLinkError: Could not load the native libraries because we encountered the following problems: no _lightgbm in java.library.path and Could not find resource /com/microsoft/ml/lightgbm/osx/x86_64/lib_lightgbm.dylib in jar. at com.microsoft.ml.spark.NativeLoader.loadLibraryByName(NativeLoader.java:62) at com.microsoft.ml.spark.LightGBMUtils$.initializeNativeLibrary(LightGBMUtils.scala:24) at com.microsoft.ml.spark.TrainUtils$.trainLightGBM(TrainUtils.scala:136) at com.microsoft.ml.spark.LightGBMClassifier$$anonfun$1.apply(LightGBMClassifier.scala:47) at com.microsoft.ml.spark.LightGBMClassifier$$anonfun$1.apply(LightGBMClassifier.scala:47) at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:196) at org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:193) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)

[imatiach-msft]

@YangChaoKiKa hi, sorry for any confusion, we currently do not support running LightGBM on mac OSX and windows. This is something that we are planning to add in the future - basically we just need to compile the binaries (dylib for mac and dll for windows) and add them to the jar that is published to maven in the corresponding /osx/x86_64/*.dylib or similar for windows folder.

[icezzzzzz]

@imatiach-msft Hi, I had also met the same error while I running on spark cluster. How to fix it?

[icezzzzzz]

[mhamilton723]

@lppsuixn perhaps try using our latest master build: Maven package uploaded, use --packages com.microsoft.ml.spark:mmlspark_2.11:0.12.dev19 and --repositories https://mmlspark.azureedge.net/maven.

Until we release 0.13

[lppsuixn]

@mhamilton723 Thanks.But the same error as before.

java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_void;
  at java.lang.Class.getDeclaredFields0(Native Method)
  at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
  at java.lang.Class.getDeclaredField(Class.java:2068)
  at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1803)
  at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)
  at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:494)
  at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:482)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)
  at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)
  at java.io.ObjectOutputStream.writeClass(ObjectOutputStream.java:1213)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1120)
  at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
  at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
  at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
  at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
  at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
  at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
  at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
  at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
  at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:295)
  at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:288)
  at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:108)
  at org.apache.spark.SparkContext.clean(SparkContext.scala:2094)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:793)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:792)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
  at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
  at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
  at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:792)
  at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:2547)
  at org.apache.spark.sql.Dataset.rdd(Dataset.scala:2544)
  at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1424)
  at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:49)
  at com.microsoft.ml.spark.LightGBMClassifier.train(LightGBMClassifier.scala:22)
  at org.apache.spark.ml.Predictor.fit(Predictor.scala:96)
  ... 52 elided
Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_void
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  ... 91 more

[Oukaishen]

face the same problem please help to solve

[imatiach-msft]

@Oukaishen what operating system are you using? Is this on a cluster or local compute? Usually this error occurs when the native lightgbm .so, .dll or .dylib (depending on OS) cannot be loaded. Could you give more information about your machine? macOS isn't supported with 0.16 release but is supported on latest master.

[Oukaishen]

hi, I am running on a cluster(yarn). The cluster is maintained by other teams. I wonder if there any guide to package .so , .dll you mentioned into the jar. So that i can simply run the jar?

[IslemBenmahammed]

face the same problem please help to solve

[yinshurman]

hi , I use the latest version of 0.18.1,and faced the same problem with submit --jars ,because the spark cluster of cloud service dosen't support by --packages, so ,what is the solultions? please help.

java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_void;
        at java.lang.Class.getDeclaredFields0(Native Method)
        at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
        at java.lang.Class.getDeclaredField(Class.java:2068)
        at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1703)
        at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:72)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:484)
        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:472)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:472)
        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:369)
        at java.io.ObjectOutputStream.writeClass(ObjectOutputStream.java:1213)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1120)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
        at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:43)
        at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:100)
        at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:342)
        at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:335)
        at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:159)
        at org.apache.spark.SparkContext.clean(SparkContext.scala:2292)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:797)
        at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:796)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
        at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:796)
        at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:2975)
        at org.apache.spark.sql.Dataset.rdd(Dataset.scala:2973)
        at org.apache.spark.sql.Dataset.reduce(Dataset.scala:1610)
        at com.microsoft.ml.spark.lightgbm.LightGBMBase$class.innerTrain(LightGBMBase.scala:90)
        at com.microsoft.ml.spark.lightgbm.LightGBMRanker.innerTrain(LightGBMRanker.scala:25)
        at com.microsoft.ml.spark.lightgbm.LightGBMBase$class.train(LightGBMBase.scala:38)
        at com.microsoft.ml.spark.lightgbm.LightGBMRanker.train(LightGBMRanker.scala:25)
        at com.microsoft.ml.spark.lightgbm.LightGBMRanker.train(LightGBMRanker.scala:25)
        at org.apache.spark.ml.Predictor.fit(Predictor.scala:118)
        at LgbRank$.main(LgbRank.scala:73)
        at LgbRank.main(LgbRank.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4.run(ApplicationMaster.scala:708)
Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_void
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

[imatiach-msft]

@yinshurman sorry about the trouble you are having. I believe you just need to include the lightgbm jar with the mmlspark jar https://mvnrepository.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib the version of the lightgbm jar will depend on the mmlspark version you are using, for you it is 2.2.350 since you are using 0.18.1

[mchl0203]

@imatiach-msft Hi, I import lightgbmlib(==2.2.400 because of probability problem) jar in osx will get this error:

A needed class was not found. This could be due to an error in your runpath. Missing class: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_int64_t;
java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_int64_t;

what should I do, Thank you~

[imatiach-msft]

@mchl0203 sorry about the trouble you are having. If you are running into the probability issue (as fixed in this PR: https://github.com/Azure/mmlspark/pull/676), and you are using the latest lightgbm jar, then you will also need to update the mmlspark code to latest master or at least to a commit after that PR. What is you mmlspark code based on? If it's 0.18.1, it won't work as there are corresponding scala changes that need to be made with the new lightgbm jar.

[mchl0203]

@imatiach-msft yes, I used 0.18.1, because the newest jar is 0.18.1 in https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark. later, I write a softmax function in rawToProbabilityInPlace, it can work. Just a moment ago, I use a mmlspark jar compiled in local osx and lightgbmlib==2.2.400 jar, it also can work. Thank you very much for your reply.

[nanffiy]

@imatiach-msft
I meet same error. work fine then include lightgbm.jar Thanks!

so, if i will use other modules of mmlspark, either include the same name jar?

[kranthi3loq]

Hi, I am facing an issue while fitting the lightGBM model in spark scala Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_long_long at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) Can you please help in getting sort out this issue

[MarsXDM]

Hi, I am facing an issue while fitting the lightGBM model in spark scala Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_long_long at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:419) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:352) Can you please help in getting sort out this issue

Do you solve this issue ? I meet this issue and download lightgbmlib v2.2.35 from https://mvnrepository.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib/2.2.350 to solve the problem com.microsoft.ml.lightgbm.SWIGTYPE_p_int64_t But this Caused by: java.lang.ClassNotFoundException: com.microsoft.ml.lightgbm.SWIGTYPE_p_long_long thrown later...

[imatiach-msft]

Hmm, it seems to be something with the native dependencies... note there is the mmlspark jar and then the lightgbm jar, as you pointed above. Both have to be in sync in terms of the version, the mmlspark jar can only work with a particular lightgbm jar right now due to breaking changes/frequent updates. This error seems to indicate that there is something wrong with the lightgbm jar, either it is missing or the wrong version. Not sure I can help with more than that unless I can get a repro of the issue somehow.

[liaozp1]

https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark how do you do,i meet the same question,can you show me how you fix the issue?thank u very much

[mhamilton723]

@liaozp1 @MarsXDM are you using spark packages install or are you manually adding jars?

The spark package install will ensure all of the versions are in sync and all dependencies are resolved. https://github.com/Azure/mmlspark#python

[liaozp1]

@imatiach-msft yes, I used 0.18.1, because the newest jar is 0.18.1 in https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark. later, I write a softmax function in rawToProbabilityInPlace, it can work. Just a moment ago, I use a mmlspark jar compiled in local osx and lightgbmlib==2.2.400 jar, it also can work. Thank you very much for your reply.

他的概率有问题如何解决的啊？求指导

[liaozp1]

@imatiach-msft I meet same error. work fine then include lightgbm.jar Thanks!

so, if i will use other modules of mmlspark, either include the same name jar?

https://github.com/Azure/mmlspark#python

@liaozp1 @MarsXDM are you using spark packages install or are you manually adding jars?

The spark package install will ensure all of the versions are in sync and all dependencies are resolved. https://github.com/Azure/mmlspark#python

i now use lightgbm2.2.400 and mmlspark0.18.1, when i use lightgbm2.2.350 and mmlspark0.18.1 then the probability is false, but now i meet a new question, "Exception in thread "main" java.lang.NoClassDefFoundError: Lcom/microsoft/ml/lightgbm/SWIGTYPE_p_int64_t"; when i using spark packages install cannot work,please help me

@liaozp1 @MarsXDM are you using spark packages install or are you manually adding jars?

The spark package install will ensure all of the versions are in sync and all dependencies are resolved. https://github.com/Azure/mmlspark#python

i 

@imatiach-msft I meet same error. work fine then include lightgbm.jar Thanks!

so, if i will use other modules of mmlspark, either include the same name jar?

can u help me?i meet the same problem

[mchl0203]

@imatiach-msft yes, I used 0.18.1, because the newest jar is 0.18.1 in https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark. later, I write a softmax function in rawToProbabilityInPlace, it can work. Just a moment ago, I use a mmlspark jar compiled in local osx and lightgbmlib==2.2.400 jar, it also can work. Thank you very much for your reply.

他的概率有问题如何解决的啊？求指导

我重新定义这个计算函数，你试试

override def rawToProbabilityInPlace(raw: linalg.Vector): linalg.Vector = {
  raw match {
    case dv: DenseVector =>
      dv.values(0) = 1.0 / (1.0 + math.exp(-dv.values(0)))
      dv.values(1) = 1.0 - dv.values(0)
      dv
    case sv: SparseVector =>
      throw new RuntimeException("Unexpected error in LightGBMClassificationModel:" +
        " raw2probabilityInPlace encountered SparseVector")
  }
}

[liaozp1]

@imatiach-msft yes, I used 0.18.1, because the newest jar is 0.18.1 in https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark. later, I write a softmax function in rawToProbabilityInPlace, it can work. Just a moment ago, I use a mmlspark jar compiled in local osx and lightgbmlib==2.2.400 jar, it also can work. Thank you very much for your reply.

他的概率有问题如何解决的啊？求指导

我重新定义这个计算函数，你试试

override def rawToProbabilityInPlace(raw: linalg.Vector): linalg.Vector = {
  raw match {
    case dv: DenseVector =>
      dv.values(0) = 1.0 / (1.0 + math.exp(-dv.values(0)))
      dv.values(1) = 1.0 - dv.values(0)
      dv
    case sv: SparseVector =>
      throw new RuntimeException("Unexpected error in LightGBMClassificationModel:" +
        " raw2probabilityInPlace encountered SparseVector")
  }
}

@imatiach-msft yes, I used 0.18.1, because the newest jar is 0.18.1 in https://mvnrepository.com/artifact/com.microsoft.ml.spark/mmlspark. later, I write a softmax function in rawToProbabilityInPlace, it can work. Just a moment ago, I use a mmlspark jar compiled in local osx and lightgbmlib==2.2.400 jar, it also can work. Thank you very much for your reply.

他的概率有问题如何解决的啊？求指导

我重新定义这个计算函数，你试试

override def rawToProbabilityInPlace(raw: linalg.Vector): linalg.Vector = {
  raw match {
    case dv: DenseVector =>
      dv.values(0) = 1.0 / (1.0 + math.exp(-dv.values(0)))
      dv.values(1) = 1.0 - dv.values(0)
      dv
    case sv: SparseVector =>
      throw new RuntimeException("Unexpected error in LightGBMClassificationModel:" +
        " raw2probabilityInPlace encountered SparseVector")
  }
}

这就相当于取了一个1/1+e-x,我知道罗辑回归是这么做的，但是数模型好像不是这么算的？是不是呢？我最开始也是看到不对，然后说是2.2.400解决了，但是就报刚才那个问题，能解答一下吗

[mchl0203]

这就相当于取了一个1/1+e-x,我知道罗辑回归是这么做的，但是数模型好像不是这么算的？是不是呢？我最开始也是看到不对，然后说是2.2.400解决了，但是就报刚才那个问题，能解答一下吗

rawToProbabilityInPlace这个函数是计算概率的；报错的问题我有点忘了怎么解的，你试试在osx本地编译一下，用编译好的包导入试试

[imatiach-msft]

FYI 0.18.1 is very old, @liaozp1 @mchl0203 please use the latest version, either the master build or rc3 (it is in different maven repo, see main page for details). Please make sure lightgbm native jar is in sync with mmlspark jar.

[rgordon]

Following the instructions on the main page, attempting to install into a DataBricks cluster: Spark 3.0.1, Scala 2.12

I set: Coordinate com.microsoft.ml.spark:mmlspark_2.11:1.0.0-rc3 Repository https://mmlspark.azureedge.net/maven

and I'm able to see the LightGBM library, but attempts to instantiate the LightGBMClassifier results in the same error: java.lang.NoClassDefFoundError: org/apache/spark/ml/util/MLWritable$class

Is there something else I need to cause a diffferent jar to load? Or is it as I'm afraid of - this is not compatible with current Spark 3 builds?

[imatiach-msft]

hi @rgordon please use the latest master release which supports spark 3.0, the rc3 release does not support it: