spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
135 stars 62 forks source link

Feature request copy data with parquet format #117

Open parisni opened 1 year ago

parisni commented 1 year ago

Currently the lib support text and avro format to ingest data into redshift see https://github.com/spark-redshift-community/spark-redshift/blob/master/src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftWriter.scala#L97 but copy now supports parquet as well. See https://docs.aws.amazon.com/redshift/latest/dg/copy-parameters-data-format.html#copy-parquet