search

CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

ParquetDataCommitter fails when performing listStatus on the output directory if it does not exist #55

Closed vavison closed 5 years ago

vavison commented 5 years ago

Expected Behavior

listStatus is performed as part of the snapshot cleanup mechanism (to identify existing snapshots). If the output directory does not exist, the list of existing snapshots should be empty

Actual Behavior

The application will fail if the output directory is empty (i.e. the first time an application is run)

alexjbush commented 5 years ago

Not an issue