woodlee
commented
6 years ago The value of the cluster argument isn't meant to be the full hostname, but rather only the name of the cluster, as seen e.g. on the Redshift console page. In your case that's probably just dev-randomrobindw.
Also, I noticed that your S3 path includes /test-parquet/ in it... just be aware that this script doesn't do Parquet. But maybe you're planning on making that transformation separately.
@woodlee
Just thought of giving a shot...
`./unload_rs --region us-east-1 --s3-bucket hdw-dstest-randomrobin-net --s3-path-prefix test_spectrify/test-parquet//redshift/a_kevin/bi2405_greencore_uni_control_aug2017/ --cluster dev-randomrobindw.cqbnnbhakqr4.us-east-1.redshift.amazonaws.com --db randomrobindw --schema a_kevin --table bi2405_greencore_uni_control_aug2017 --db-user randomrobindw --iam-role-arn 'arn:aws:iam::196711603480:role/redshift-hdw-users-s3'
MainThread) 2018-03-25 19:22:12 INFO [main:125] Running with options: {'audit_col': None, 'audit_max_val': None, 'audit_min_val': None, 'cluster': 'dev-randomrobindw.cqbnnbhakqr4.us-east-1.redshift.amazonaws.com', 'db': 'hauteloodw', 'db_user': 'randomrobindw', 'delete_on_success': False, 'iam_role_arn': 'arn:aws:iam::196711603480:role/redshift-hdw-users-s3', 'keep_on_failure': False, 'region': 'us-east-1', 's3_bucket': 'hdw-dstest-randomrobin-net', 's3_path_prefix': 'test_spectrify/test-parquet//redshift/a_kevin/bi2405_greencore_uni_control_aug2017/', 'schema': 'a_kevin', 'table': 'bi2405_greencore_uni_control_aug2017'} Traceback (most recent call last): File "./unload_rs", line 419, in
main(opts)
File "./unload_rs", line 184, in main
sqlalchemy_engine = get_sqlalchemy_engine(opts.region, opts.cluster, opts.db, opts.db_user)
File "./unload_rs", line 161, in get_sqlalchemy_engine
desc_response = client.describe_clusters(ClusterIdentifier=cluster)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidParameterValue) when calling the DescribeClusters operation: The parameter DBInstanceIdentifier is not a valid identifier. Identifiers must begin with a letter; must contain only ASCII letters, digits, and hyphens; and must not end with a hyphen or contain two consecutive hyphens.
`