search

sadikovi / spark-netflow

NetFlow data source for Spark SQL and DataFrames
Apache License 2.0
18 stars 11 forks source link

Spark 3.0 support #79

Closed alexott closed 3 years ago

alexott commented 3 years ago

Spark 3.0 was released this year with a lot of optimizations, and improvements. But the code isn't compiled out of the box due of the internal changes in the Spark, etc. The amount of changes is relatively small although, and I have them in my fork. (Unfortunately, because of the problems with sbt and expired certificates, I was compiled the code using Maven instead).

P.S. I have branch for 2.4 as well, but there changes aren't so big.

sadikovi commented 3 years ago

Thanks for reporting the issue. That's right, the package would need to be fixed for Spark 3.0. I will update the code based on your changes!

sadikovi commented 3 years ago

2.1.0 version is available with Spark 3.0.x support, see https://spark-packages.org/package/sadikovi/spark-netflow.