search

pgspider / parquet_s3_fdw

ParquetS3 Foreign Data Wrapper for PostgresSQL
Other
203 stars 28 forks source link

Can it load gzipped parquet files? #27

Open anentropic opened 7 months ago

anentropic commented 7 months ago

AWS has an option to export an RDS db snapshot to parquet files in an S3 bucket. But the resulting files are gzipped

Is it possible to directly load them with the fdw, or I need to run a batch job to unzip them first?

Maybe this is a dumb question, the files are like part-00000-fa89fa59-af57-4db3-81f3-bba1597c6e56-c000.gz.parquet... is that just a parquet file with internal gzip compression? Can the fdw read that?

hrkuma commented 6 months ago

Hello,

Maybe you can access such kind of file after merging https://github.com/pgspider/parquet_s3_fdw/pull/25 .

lamdn1409 commented 3 weeks ago

@anentropic

parquet_s3_fdw does not support GZIP compression method. It supports the following methods: SNAPPY, ZSTD and UNCOMPRESSED.