I try to use the appendFile method of ParquetFileWriter to merge some smaller Parquet files into one large parquet file. After I finished the merge, I tried deleting the smaller files, but couldn't actually delete them.
After reading the ParquetFileWriter source code, I found that the file he opened was not closed in the end method, which seemed unreasonable.
When testing on Windows, it was found that these implicitly opened files cannot be deleted or renamed. This seems to be related to Windows file handles. But it's worked fine on macOS.
Anyway, I will submit a patch to ensure that files opened through the appendFile function are closed correctly.
I try to use the
appendFilemethod ofParquetFileWriterto merge some smaller Parquet files into one large parquet file. After I finished the merge, I tried deleting the smaller files, but couldn't actually delete them.After reading the ParquetFileWriter source code, I found that the file he opened was not closed in the end method, which seemed unreasonable.
When testing on
Windows, it was found that these implicitly opened files cannot be deleted or renamed. This seems to be related to Windows file handles. But it's worked fine on macOS.Anyway, I will submit a patch to ensure that files opened through the appendFile function are closed correctly.
Reporter: Deng An
PRs and other links:
Note: This issue was originally created as PARQUET-2477. Please see the migration documentation for further details.