search

dmlc / xgboost

Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow
https://xgboost.readthedocs.io/en/stable/
Apache License 2.0
26.34k stars 8.73k forks source link

[JVM-Package] Consider updating kryo to 2.22 in xgboost4j #4548

Closed danielhstahl closed 5 years ago

danielhstahl commented 5 years ago

When I use xboost4j in Spark 2.3.0 or 2.3.3, I get the following warning message:

Warning: Local jar /root/.ivy2/jars/com.esotericsoftware.reflectasm_reflectasm-1.07.jar does not exist, skipping.

When connecting over Livy, this warning becomes and error and I can't continue. I believe that this issue is due to Kryo 2.21 and is fixed in Kryo 2.22.

CodingCat commented 5 years ago

is there any references to this error?

danielhstahl commented 5 years ago

The error is

ERROR spark.SparkContext: Error initializing SparkContext. java.io.FileNotFoundException: File file:[myhome]/.ivy2/jars/com.esotericsoftware.reflectasm_reflectasm-1.07.jar does not exist

CodingCat commented 5 years ago

I mean why it is related to 2.21 v.s. 2.22?

danielhstahl commented 5 years ago

Great question! I had thought I had read that this issue was fixed in 2.22 but I can't find where I read it now.

Have you had this warning show in testing? I'd be happy to run a test against several versions of kryo to see if it helps.

danielhstahl commented 5 years ago

I was able to get around the error by adding kryo 2.22 to my spark-defaults.conf:

spark.jars.packages=ml.combust.mleap:mleap-spark-extension_2.11:0.13.0,ml.combust.mleap:mleap-xgboost-spark_2.11:0.13.0,ml.dmlc:xgboost4j-spark:0.82,com.esotericsoftware.kryo:kryo:2.22

So it does seem like kryo 2.22 would fix the issue.

danielhstahl commented 5 years ago

And I just confirmed in a docker image using apache spark:

./spark-shell --packages=ml.combust.mleap:mleap-spark-extension_2.11:0.13.0,ml.combust.mleap:mleap-xgboost-spark_2.11:0.13.0,ml.dmlc:xgboost4j-spark:0.82,com.esotericsoftware.kryo:kryo:2.22

Does not give me the warning.

CodingCat commented 5 years ago

sure, welcome to file a PR for this

danielhstahl commented 5 years ago

Opened a PR.