The check if (dir.exists()) in line 179 above may cause subsequent partition write failed if partitions are more than 2 in the test below. Because partitions are written in a map in line 211 above, subsequent partition writes would fail when checking if (dir.exists()).
https://github.com/tensorflow/ecosystem/blob/791a42f427139f2cb7bbae8a93a9e666a38c2bcc/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/DefaultSource.scala#L173-L215
The check
if (dir.exists())in line 179 above may cause subsequent partition write failed if partitions are more than 2 in the test below. Because partitions are written in a map in line 211 above, subsequent partition writes would fail when checkingif (dir.exists()).https://github.com/tensorflow/ecosystem/blob/791a42f427139f2cb7bbae8a93a9e666a38c2bcc/spark/spark-tensorflow-connector/src/test/scala/org/tensorflow/spark/datasources/tfrecords/LocalWriteSuite.scala#L42-L70
The exception thrown should be similar to https://github.com/tensorflow/ecosystem/pull/141#issuecomment-542353106