Glue get_tables fails to return tables if QueryAsOfTime parameter is specified

GabrielEisenbergOlympus commented 1 week ago

Describe the bug

When calling get_tables without QueryAsOfTime being specified, tables are returned, however, when it is specified with either a datetime (Python native or with Pendulum), no tables are returned. Funny enough, this parameter works perfectly in get_table.

Expected Behavior

Obtain a list of tables, matching the input Expression for a given point in time as specified with QueryAsOfTime.

Current Behavior

The response contains an empty TableList:

{'TableList': [], 'ResponseMetadata': {'RequestId': 'FAKED-REQUEST-ID', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Fri, 21 Jun 2024 07:54:34 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '16', 'connection': 'keep-alive', 'x-amzn-requestid': 'FAKED-REQUEST-ID'}, 'RetryAttempts': 0}}

Reproduction Steps

import boto3
import pendulum as pd

client = boto3.client("glue", region_name="af-south-1") # Perhaps the region contributes to this issue, hence the specification.
query_as_of_time = pd.datetime(
    year=2024, month=6, day=14, hour=0, minute=54, second=0, tz="Africa/Johannesburg"
) # Experience the same behavior with datetime.datetime or a timestamp equivalent value

response = client.get_tables(
        DatabaseName="glue_database",
        Expression=f"^glue_table$",
        QueryAsOfTime=query_as_of_time,
    )

print(response)

Possible Solution

Mimic the use of the parameter in get_table as this works perfectly.

Additional Information/Context

I have also tried adding a capture group to Expression which did not change the response when QueryAsOfTime was set,

SDK version used

1.34.131 on Python 3.11.9

Environment details (OS name and version, etc.)

Windows 11

tim-finnigan commented 6 days ago

Thanks for reaching out. I similarly do not see results when specifying QueryAsOfTime. The Boto3 get_tables command involves a call to the underlying GetTables API, so I'll need to reach out to the Glue team for more information here.

Here is the API documentation for this parameter:

And the Boto3 docs use QueryAsOfTime=datetime(2015, 1, 1) as an example. I tested with datetime in us-west-2 some I'm not sure the region is a factor here:

import boto3
from datetime import datetime

client = boto3.client("glue")

response = client.get_tables(
    DatabaseName="gluedb",
    QueryAsOfTime=datetime(2023, 1, 1)
)

print(response)

When not specifying QueryAsOfTime, the datetime values in the results look like this:

datetime.datetime(2023, 1, 1, 1, 6, tzinfo=tzlocal())

So there appears to be some inconsistency here. I'll let you know when I have updates from the Glue team, and please refer to https://github.com/aws/aws-sdk/issues/768 for updates going forward.

github-actions[bot] commented 6 days ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

boto / boto3