search

hellonarrativ / spectrify

Export Redshift data and convert to Parquet for use with Redshift Spectrum or other data warehouses.
https://aws.amazon.com/blogs/big-data/narrativ-is-helping-producers-monetize-their-digital-content-with-amazon-redshift/
MIT License
116 stars 25 forks source link

Command-Line export problem #34

Closed ChristianPAC closed 5 years ago

ChristianPAC commented 6 years ago

Description

I'm trying Export Redshift table XXX to a folder of CSV files on S3 from Command-line Usage but I always have an error...

What I Did

In CMD: spectrify --(name).us-east-1.redshift.amazonaws.com --user=(XXX)--db=(XXX) export reporting.delivery_method \'s3://(name)/reporting.delivery_method'

ERROR: sqlalchemy.exc.InternalError: (psycopg2.InternalError) UNLOAD destination is not supported. (Hint: only S3 based unload is allowed) [SQL: "\n UNLOAD ('select * from reporting.delivery_method')\n to %(s3_path)s\n CREDENTIALS %(credentials)s\n ESCAPE MANIFEST GZIP ALLOWOVERWRITE\n MAXFILESIZE 1 gb;\n "] [parameters: {'s3_path': "\'s3://(erase)/reporting.delivery_method'/csv/", 'credentials': 'aws_access_key_id=(erase);aws_secret_access_key=(erase)'}] (Background on this error at: http://sqlalche.me/e/2j85)

Idk what is the problem... maybe the sintaxis from "'s3://example-bucket/my_table'", But idk I read this: US East (N. Virginia) Region endpoint, http://s3.amazonaws.com/bucket on https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingBucket.html#access-bucket-intro and I did that... (I'm using Virginia)

'http://s3.amazonaws.com/XXX/reporting.delivery_method' or 's3://XXX/reporting.delivery_method'

I'm using the right host, user, db

HELP ME!!!!!!!!!!

c-nichols commented 6 years ago

Have you tried it without the single quotes around the s3 path?

My knowledge of Windows is limited but I do not believe Cmd.exe is compatible with quoted arguments

ChristianPAC commented 6 years ago

Thank you for answetring me...

I already tried that, but it appears the same...

sqlalchemy.exc.InternalError: (psycopg2.InternalError) UNLOAD destination is not supported. (Hint: only S3 based unload is allowed)

c-nichols commented 5 years ago

@ChristianPAC did you ever get anywhere with this?

I'm guessing it's got something to do with WIndows vs. Mac/Linux... spectrify uses click which is supposed to paper over most of those differences but there could be an issue with parsing s3 paths or something else.