Closed DrYSG closed 2 years ago
Hi,
Thanks for reporting this, but unfortunately Spark 3.0 is not supported by HandySpark. Spark 3.0 introduced a lot of changes, that's why these errors are showing up to you. Some of the functionalities of HandySpark are covered by Databricks' own Koalas package.
Best, Daniel
Using DataBricks v7.4 ML cluster, with Spark 3.01
(also I think you are using MultiClassMetrics to get the AUC value, but that seems to be high when one has a LR model. See: https://stackoverflow.com/questions/60772315/how-to-evaluate-a-classifier-with-apache-spark-2-4-5-and-pyspark-python )
I get this error when trying to do the confusion matrix:
I also get deprecation warnings when doing the ROC plots: