pola-rs / polars

Dataframes powered by a multithreaded, vectorized query engine, written in Rust
https://docs.pola.rs
Other
30.25k stars 1.95k forks source link

Polars write database - Rest API call failing with AWS Lambda trigger #17880

Open Sankartk opened 3 months ago

Sankartk commented 3 months ago

Checks

Reproducible example

I'm using polars scan_parquet and write_database in my application component. I'm having the if_table_exists parameter in the write database as append. I have a REST API endpoint setup for this. When I post to the endpoint locally, it's working. But when I try to call it from my AWS Lambda fn, it's failing. But what's weird is I'm seeing an error message that it's trying to create tables rather than append to it. There's no reference to create tables in my code. As said, I used the write_database method from polars. Has anyone faced this error? Please let me know if any suggestions/fix for this

Log output

No response

Issue description

I'm using polars scan_parquet and write_database in my application component. I'm having the if_table_exists parameter in the write database as append. I have a REST API endpoint setup for this. When I post to the endpoint locally, it's working. But when I try to call it from my AWS Lambda fn, it's failing. But what's weird is I'm seeing an error message that it's trying to create tables rather than append to it. There's no reference to create tables in my code. As said, I used the write_database method from polars. Has anyone faced this error? Please let me know if any suggestions/fix for this

Expected behavior

Instead of direct insert, it's trying to create a table. In the table name parameter I have schema.table_name

Installed versions

polars == 1.1.0

deanm0000 commented 2 months ago

Could it be that locally you've got some environment variables steering what you're doing that aren't matched on lambda?