Open Jasonxdy opened 1 month ago
Hey,
I haven't been able to recreate this issue. After the temp table is created, it's actually referenced in the Athena query here. So if the temp table has indeed not been created in time, I would expect the Athena query to fail before the Glue table deletion. Secondly, the Glue table deletion method does check if the table exists, so it's supposed to skip the deletion step if it can't find the table.
Can you please share a stack trace for when this error occurs, along with the Python and AWS SDK for pandas versions that you are using?
Best regards, Leon
Describe the bug
When calling to_iceberg function, I found out that some of the temp table (ex. temp_table_xxxxx) created by the function persists in Athena. Upon checking the CloudTrail, the DeleteTable glue API has failed to delete the temp table with
EntityNotFoundException
. The CreateTable API and DeleteTable API was called nearly at the same time, and since the CreateTable API is async, I assume that the DeleteTable API failed because the temp table was not created yet.I suggest to add a logic to check whether table was created or not before calling DeleteTable API in to_iceberg function.
How to Reproduce
Expected behavior
The temp_table_xxxxx has to be deleted properly in Athena
Your project
No response
Screenshots
No response
OS
Window
Python version
-
AWS SDK for pandas version
-
Additional context
No response