search

sramirez / spark-MDLP-discretization

Spark implementation of Fayyad's discretizer based on Minimum Description Length Principle (MDLP)
Apache License 2.0
44 stars 27 forks source link

AbstractMethodError in Spark 2.3.0 #38

Open Majid-Soheili opened 5 years ago

Majid-Soheili commented 5 years ago

I would like to discretize the Epsilon dataset that it has 2k features and 400k records. To achieve this, I utilized the Spark 2.3.0. Once I execute the code, I deal with below error. It is noteworthy this error had not occurred when I used Spark 2.2.0, but It seems that the Spark 2.2.0 is not suitable for a high dimensional dataset. If is it possible, please fix this problem.

######################################################### Exception in thread "main" java.lang.AbstractMethodError at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99) at org.apache.spark.mllib.feature.MDLPDiscretizer.initializeLogIfNecessary(MDLPDiscretizer.scala:48) at org.apache.spark.internal.Logging$class.log(Logging.scala:46) at org.apache.spark.mllib.feature.MDLPDiscretizer.log(MDLPDiscretizer.scala:48) at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) at org.apache.spark.mllib.feature.MDLPDiscretizer.logInfo(MDLPDiscretizer.scala:48) at org.apache.spark.mllib.feature.MDLPDiscretizer.runAll(MDLPDiscretizer.scala:106) at org.apache.spark.mllib.feature.MDLPDiscretizer$.train(MDLPDiscretizer.scala:335) at org.apache.spark.ml.feature.MDLPDiscretizer.fit(MDLPDiscretizer.scala:149) #########################################################

jfgosselin commented 5 years ago

+1 I ran into the same issue today . Is there any plan for an upgrade to Spark 2.3.0 ?

jfgosselin commented 5 years ago

I've recompiled spark-MDLP-discretization with Spark 2.3.2 to and Scala 2.11.2 to solve the issue.

Majid-Soheili commented 5 years ago

I've recompiled spark-MDLP-discretization with Spark 2.3.2 and Scala 2.11.2 to solve the issue.

Hi Jean, If it is possible for you, share your compiled library. Also, there is another issue on the last version on the library that is so important in my impression, The issue and the solution were reported in the address below.

https://github.com/sramirez/spark-MDLP-discretization/issues/36

If it is possible for you, please fix the issue and share it. Thanks a lot.