Add pandasql and duckdb packages for python3.12

[keithrozario]

Pandassql didn't work for some reason. But duckdb was successful.

arn:aws:lambda:af-south-1:770693421928:layer:Klayers-p312-duckdb:1

[LyleMartin]

Thank you so much! Sent from my Verizon, Samsung Galaxy smartphone -------- Original message --------From: Keith Rozario @.> Date: 6/15/24 4:41 AM (GMT-05:00) To: keithrozario/Klayers @.> Cc: LyleMartin @.>, Author @.> Subject: Re: [keithrozario/Klayers] Add pandasql and duckdb packages for python3.12 (PR #419) Pandassql didn't work for some reason. But duckdb was successful. arn:aws:lambda:af-south-1:770693421928:layer:Klayers-p312-duckdb:1

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>

[LyleMartin]

Thank you. When I add the library I get no responses from the Lambda function. I have an exception set up to to SNS but also a dead letter queue. So usually on exception I get a message.

I tried changing the order of the import calls, and the order of merging the libraries and still nothing. Only if I take layer out do I get an error that the module could not be imported.

When I use your layers for pandas, numpy and now duckdb it goes over the size limit. When I use the AWS layer for pandas and your layer for duckdb it fits in the limit. But when I run the code I get no response at all -- no exception, no results. No logging either, so I know it is dying.

I am at a loss. I don't want to go to containers. Any thoughts are greatly appreciated.

Thanks,

-Lyle

---

From: "Keith Rozario" @.> Sent: 6/15/24 4:42 AM To: keithrozario/Klayers @.> Cc: LyleMartin @.>, Author @.> Subject: Re: [keithrozario/Klayers] Add pandasql and duckdb packages for python3.12 (PR #419)

Pandassql didn't work for some reason. But duckdb was successful. arn:aws:lambda:af-south-1:770693421928:layer:Klayers-p312-duckdb:1 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

[keithrozario]

No response at all? Nothing even in Cloudwatch logs that you can validate?

That's strange.

Try importing duckdb only first. And see if there's anything wrong with that layer.

Lambda has an init duration limit of 10 seconds, and if the function takes longer than that to initialize it will fail. But even then I thought it would still log out something you can see in Cloudwatch.

Try the following:

1. Import only duckdb and see if it works.
2. Increase the memory of lambda to improve the performance. Try around 2GB.
3. If you get duckdb to work, then slowly work backwards to includes pandas.

Hope that helps.

[keithrozario]

Oh, one more thing. If nothing works try importing duckdb within your function handler -- instead of before it. This forces it to run during runtime instead of init time. And that might help.

[LyleMartin]

Thank you

---

From: "Keith Rozario" @.> Sent: 6/17/24 7:53 PM To: keithrozario/Klayers @.> Cc: LyleMartin @.>, Author @.> Subject: Re: [keithrozario/Klayers] Add pandasql and duckdb packages for python3.12 (PR #419)

Oh, one more thing. If nothing works try importing duckdb within your function handler -- instead of before it. This forces it to run during runtime instead of init time. And that might help. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.