search

titicaca / spark-iforest

Isolation Forest on Spark
Apache License 2.0
226 stars 90 forks source link

Parameter approxQuantileRelativeError not usable when running with Spark 3 #36

Closed JJorczik closed 2 years ago

JJorczik commented 2 years ago

Setting approxQuantileRelativeError using Scala or Python API of branch spark3 returns the following error. Is there an easy way to fix this? Thank you in advance!

java.lang.UnsupportedOperationException: The default jsonEncode only supports string, vector and matrix. org.apache.spark.ml.param.Param must override jsonEncode for java.lang.Double.
  at org.apache.spark.ml.param.Param.jsonEncode(params.scala:100)
  at org.apache.spark.ml.util.Instrumentation.$anonfun$logParams$2(Instrumentation.scala:109)
  at scala.Option.map(Option.scala:230)
  at org.apache.spark.ml.util.Instrumentation.$anonfun$logParams$1(Instrumentation.scala:106)
  at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
  at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
  at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
  at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:38)
  at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
  at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
  at scala.collection.AbstractTraversable.flatMap(Traversable.scala:108)
  at org.apache.spark.ml.util.Instrumentation.logParams(Instrumentation.scala:105)
  at org.apache.spark.ml.iforest.IForest.$anonfun$fit$1(IForest.scala:503)
  at org.apache.spark.ml.util.Instrumentation$.$anonfun$instrumented$1(Instrumentation.scala:191)
  at scala.util.Try$.apply(Try.scala:213)
  at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:191)
  at org.apache.spark.ml.iforest.IForest.fit(IForest.scala:495)
  ... 47 elided
titicaca commented 2 years ago

It looks like the default parameter jsonEncode has changed and doesn't fit for double .. I think you can have a try by converting the param type from double to str..