Hi @djuang1 ,
We have tried running the poc by upgrading the parquet connector version to v1.0.28.PFB findings:
We were successfully able to process the 12MB (180485 records) snappy.parquet file.
Tried running .parquet extension file with 8MB (198603 records) and above but observed jvm restart issue.
As the number of records increases we might face a jvm restart issue and hence would be difficult to conclude on the maximum file size that will be successfully processed.
Hi @djuang1 , We have tried running the poc by upgrading the parquet connector version to v1.0.28.PFB findings: We were successfully able to process the 12MB (180485 records) snappy.parquet file. Tried running .parquet extension file with 8MB (198603 records) and above but observed jvm restart issue. As the number of records increases we might face a jvm restart issue and hence would be difficult to conclude on the maximum file size that will be successfully processed.