search

pgspider / parquet_s3_fdw

ParquetS3 Foreign Data Wrapper for PostgresSQL
Other
180 stars 24 forks source link

Can it load gzipped parquet files? #27

Open anentropic opened 2 months ago

anentropic commented 2 months ago

AWS has an option to export an RDS db snapshot to parquet files in an S3 bucket. But the resulting files are gzipped

Is it possible to directly load them with the fdw, or I need to run a batch job to unzip them first?

Maybe this is a dumb question, the files are like part-00000-fa89fa59-af57-4db3-81f3-bba1597c6e56-c000.gz.parquet... is that just a parquet file with internal gzip compression? Can the fdw read that?

hrkuma commented 1 month ago

Hello,

Maybe you can access such kind of file after merging https://github.com/pgspider/parquet_s3_fdw/pull/25 .