Facing issues with json4s package, while saving model. Also not able to create a fat jar due to version conflict between liberaries.

[sandeep1288]

I downloaded the iForest jar from : https://dl.bintray.com/linkedin/maven/com/linkedin/isolation-forest/isolation-forest_2.11/0.2.2/ But while trying to save model, I'm getting below options. I tried multiple versions of json4s modules, but still not able to fix the error. java.lang.NoSuchMethodError: org.json4s.jackson.JsonMethods$.parse(Lorg/json4s/JsonInput;Z)Lorg/json4s/JsonAST$JValue;

Apart from this I tried to compile iForest using the mentioned steps, but that didn't work, and threw version conflict errors. Is it possible for you to share the detailed info, which've been used to compile the package, and versions of modules being used.

[sandeep1288]

Below function threw error: private def getMetadataToSave( instance: Params, spark: SparkSession, extraMetadata: Option[JObject] = None): String = {

  val uid = instance.uid
  val cls = instance.getClass.getName
  val params = instance.extractParamMap.toSeq
  val jsonParams = render(params.map { case ParamPair(p, v) =>
    p.name -> parse(p.jsonEncode(v))
  }.toList)
  val basicMetadata = ("class" -> cls) ~
    ("timestamp" -> System.currentTimeMillis()) ~
    ("sparkVersion" -> spark.sparkContext.version) ~
    ("uid" -> uid) ~
    ("paramMap" -> jsonParams)
  val metadata = extraMetadata match {
    case Some(jObject) => basicMetadata ~ jObject
    case None => basicMetadata
  }
  val metadataJson = compact(render(metadata))

  metadataJson
}

}

And problem is happening while saving model.

[jverbus]

What version of Spark and Scala are you using?

The recommended versions for the project are:

Scala 2.11.8 Spark 2.3.0

These are the versions used to produce the jars on Bintray.

[sandeep1288]

I'm using databrick platform, and below is the scala and spark version: spark-version : 2.4.3 scala-version : 2.11.12

In order to make current iForest working on latest spark version, what are the essential steps which I can follow. There should not be much difference between spark 2.3 and 2.4, apart from json4s.

[jverbus]

It is currently recommended to use Scala 2.11 and Spark 2.3.

You will need to build the isolation-forest library yourself using Spark 2.4 if you want to use it in a Spark 2.4 environment. You will need to change some dependencies in the module-level build.gradle file.

Please try the instructions in this ticket: https://github.com/linkedin/isolation-forest/issues/1

[sandeep1288]

Thanks a lot. I'll give it a try.

[jverbus]

@sandeep1288: Cool. Please let me know if it works. Thanks!

[sandeep1288]

Thanks a lot, I'm able to make it working with latest versions of spark.

[sandeep1288]

It worked actually. Thanks a lot for your help.

Sent from Yahoo Mail on Android

On Wed, 18 Sep 2019 at 12:57, James Verbusnotifications@github.com wrote:
@sandeep1288: Cool. Please let me know if it works. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.