Open NirTatcher opened 1 month ago
Thanks so much @NirTatcher! This looks good to me but I'd love it if someone with more Redshift knowledge gave it a look too. Maybe @austinweisgrau or @Jason94?
Thanks so much @NirTatcher! This looks good to me but I'd love it if someone with more Redshift knowledge gave it a look too. Maybe @austinweisgrau or @Jason94?
Sure thing @shaunagm! If it would make it easier I can just make all of that in separated functions so like unload (as text by default), unload as csv, unload as json, unload as parquet functions. Also, I believe it's much more powerful (at least for my use) to unload from redshidt to s3 as csv/json/parquert than in text format. So I'll be waiting and would be happy to receive any feedback from @Jason94 or @austinweisgrau so thank you all in advance!
format
parameter to the unload from redshift to S3 function. We can use either CSV or JSON or PARQUET (highly recommended for large tables, using PARQUET files are smaller than CSVs, can be read and written much faster compared to CSV), currently we only enable to unload as a TXT file by default.ADDQUOTES
.DELIMITER
,ADDQUOTES
,ESCAPE
,NULL AS
,HEADER
,GZIP (compression)
options.DELIMITER
,HEADER
,ADDQUOTES
,ESCAPE
,NULL AS
options.Hope this will be beneficial to anyone except me and pass all of the live tests. @shaunagm please let me know if there are any changes that need to be made to get this implemented. Thank you! More on PARQUET here. More on UNLOAD here.