move-coop / parsons

A python library of connectors for the progressive community.
Other
254 stars 125 forks source link

[Enhancement] - Enable Unloading Redshift Tables to S3 in either JSON, PARQUET or CSV format #1052

Open NirTatcher opened 1 month ago

NirTatcher commented 1 month ago
image

Hope this will be beneficial to anyone except me and pass all of the live tests. @shaunagm please let me know if there are any changes that need to be made to get this implemented. Thank you! More on PARQUET here. More on UNLOAD here.

shaunagm commented 1 month ago

Thanks so much @NirTatcher! This looks good to me but I'd love it if someone with more Redshift knowledge gave it a look too. Maybe @austinweisgrau or @Jason94?

NirTatcher commented 1 month ago

Thanks so much @NirTatcher! This looks good to me but I'd love it if someone with more Redshift knowledge gave it a look too. Maybe @austinweisgrau or @Jason94?

Sure thing @shaunagm! If it would make it easier I can just make all of that in separated functions so like unload (as text by default), unload as csv, unload as json, unload as parquet functions. Also, I believe it's much more powerful (at least for my use) to unload from redshidt to s3 as csv/json/parquert than in text format. So I'll be waiting and would be happy to receive any feedback from @Jason94 or @austinweisgrau so thank you all in advance!