search

linkedin / spark-tfrecord

Read and write Tensorflow TFRecord data from Apache Spark.
BSD 2-Clause "Simplified" License
290 stars 57 forks source link

fix OutputWriter break in spark 3.2 #37

Closed tangyl closed 2 years ago

tangyl commented 2 years ago

see https://github.com/apache/spark/commit/7f51106c0df190b3364848f8808e03105da63681

junshi15 commented 2 years ago

Thanks for your PR. Do you know if this works with Spark 3.0?

tangyl commented 2 years ago

It works for 3.0. I passed test with spark 3.0.0 and spark 3.2.0

mvn -Pscala-2.12 clean test -Dspark.version=3.0.0

mvn -Pscala-2.12 clean test -Dspark.version=3.2.0