search

linkedin / spark-tfrecord

Read and write Tensorflow TFRecord data from Apache Spark.
BSD 2-Clause "Simplified" License
290 stars 57 forks source link

encountered a FileAlreadyExistsException when trying to save array of arrays of arrays (3d array) #35

Open GraceHLiu opened 3 years ago

GraceHLiu commented 3 years ago

I got a FileAlreadyExistsException when I was trying to save a pyspark dataframe that has 3d array columns. The saving worked for me when I only had 1d, 2d, and scalars in the dataframe. Is there any plan to support 3d arrays in the near future?

junshi15 commented 3 years ago

Thanks for your feedback. We don't have a plan to support 3D array.