chrismld / aws-lambda-redshift-copy

AWS Lambda function that runs the copy command into Redshift
40 stars 18 forks source link

Unable to fetch key value #1

Open Rammmsankar opened 6 years ago

Rammmsankar commented 6 years ago

Hi

CloudWatch shows the following error :

An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.: NoSuchKey Traceback (most recent call last): File "/var/task/copy.py", line 43, in handler raise e NoSuchKey: An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.

Appreciate if you could help.

Regards Ramu

Rammmsankar commented 6 years ago

Resolved it by replacing s3 = boto3.client('s3') with s3client = boto3.client('s3', endpoint_url='s3-external-1.amazonaws.com')

but now facing another issue as below Unable to import module 'copy': /var/task/psycopg2/_psycopg.so: undefined symbol: _Py_ZeroStruct

dre2004 commented 6 years ago

Did you find a work around? I've got the same issue (/var/task/psycopg2/_psycopg.so: undefined symbol: _Py_ZeroStruct

chrismld commented 6 years ago

@Rammmsankar @dre2004 could you please upload a screenshot of the folder/files structure of the zip file you're uploading to AWS Lambda? Also the configuration, especially for the handler. Did you choose v 2.7 for Python?

sureshbabumandava commented 6 years ago

@christianhxc , i get same issue now /var/task/psycopg2/_psycopg.so: undefined symbol: _Py_ZeroStruct .. can you please let me know what was the fix for this?

AakashBasu commented 5 years ago

Hey @christianhxc , I am running into the similar error (using Python 3.6 in AWS Lambda):

Unable to import module 'copy': /var/task/psycopg2/_psycopg.so: undefined symbol: _Py_ZeroStruct

I've commented most of your code and simply want to experiment a select * from the Redshift DB by connecting with it. SS follows:

image

AakashBasu commented 5 years ago

I used this repo to solve the issue:

https://github.com/jkehler/awslambda-psycopg2

But now stuck at:

could not translate host name "jdbc:redshift://redshift-cluster-abc.grga543tf.us-west-1.redshift.amazonaws.com" to address: Name or service not known

I think it is an issue with the string I am passing and can be solved. Still if anyone came across this problem, do help.

chrismld commented 5 years ago

@AakashBasu I'm glad it worked by using the other code you found. I'm planning on updating the code by using lambda layers, stay tuned.

Now, regarding the connection string, it looks like the ":redshift:" is the problem. The way I connect to Redshif is by using the below code where I basically build the connection string:

conn = psycopg2.connect("dbname=" + db_database
                                + " user=" + db_user
                                + " password=" + db_password
                                + " port=" + db_port
                                + " host=" + db_host)