Fix compile for Spark 2.4 SNAPSHOT and only catch NonFatal

mengxr commented 6 years ago

The build is currently broken with Spark 2.4. This PR fixes the compile. It won't pass CI since 2.4.0 SNAPSHOT is not available on maven. I will leave the PR here until RC is out.

only catch NonFatal instead of Throwable. Our job hanged on Spark 2.4 cluster due to class compatibility issue. I'm not sure if catching Throwable is the root cause here. Let me know if we do want to catch Throwable.
Remove afterBenchmark from MLlib. The API no longer exists in Spark.
Update tree generation code, because it uses some internal APIs that we changed in 2.4.

cc: @MrBago @juliuszsompolski

mengxr commented 6 years ago

Run succeeded with the new jar: https://dogfood.staging.cloud.databricks.com/#job/830162/run/80.

juliuszsompolski commented 6 years ago

cc @npoggi

mengxr commented 6 years ago

@juliuszsompolski @npoggi I added Apache snapshot repo to resolvers. Shall we merge this patch? It will help 2.4 QA.

databricks / spark-sql-perf

Fix compile for Spark 2.4 SNAPSHOT and only catch NonFatal #164