Closed fliverance closed 2 years ago
After running some tests, it appears that the Lake Formation engine does not return work units for an empty Glue table.
To reproduce:
import awswrangler as wr
wr.catalog.create_parquet_table(
database="my_db",
table="my_empty_table",
path="s3://my_bucket/my_prefix",
table_type="GOVERNED",
columns_types={"name": "string", "value": "bigint"}
)
above command effectively creates an empty table in the AWS Glue catalog
Then when running a Lake Formation query against this table and attempting to get the work units from the engine, none are returned. Without work units, an arrow table cannot be obtained and the schema cannot be inferred. Thus an empty list is returned instead of a dataframe
Describe the bug When using wrangler to write a parquet file to a table, it's possible for the upload to fail (i.e. if network is interrupted or something). This can cause lf catalog to appear that it has the S3 file, but it does not actually exist. When the table is subsequently read, it returns an empty DF instead of throwing an exception, leading to silent failures/data corruption/etc. Wrangler should be able to identify that the LF parquet file was not actually found, and should fail fast and throw an exception instead.
To Reproduce Steps to reproduce the behavior. Also add details about Python and Wrangler's version and how the library was installed.
VENV setup:
Example pseudocode:
P.S. Don't attach files. Please, prefer add code snippets directly in the message body.