[CT-2499] [Bug] dbt docs generate (>1.3) does not work with glue catalog with large number of tables in schema

[josephberni]

Is this a new bug in dbt-core?

• [X] I believe this is a new bug in dbt-core
• [X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Currently when attempting to run dbt docs generate (dbt=1.4.6) you get a failure message if you have a large number of tables in any given schema. This is because the package will attempt to run the following code show table extended in <schema_name> like '<table_1>|<table_2>|...'.

The call to glue has the following constraint 'Length Constraints: Minimum length of 0. Maximum length of 2048.' which results in the query failing and ultimately an inability to create dbt docs when using glue on any version above 1.3

Expected Behavior

dbt docs generate to work regardless of how many tables you have in a given schema.

Steps To Reproduce

• Use glue as your metastore
• Have a large number of tables in a given schema
• Attempt to run dbt docs generate

Relevant log output

failed to satisfy constraint: Member must have length less than or equal to 2048 (Service: AWSGlue; Status Code: 400; Error Code: ValidationException; Request ID: <REQUEST_ID>; Proxy: null))

Environment

- Python: 3.8.10
- dbt: 1.4.6

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

dbt-databricks==1.4.2

[jtcohen6]

Hey @josephberni, thanks for opening!

My sense here is that this logic should live inside dbt-databricks, and do something to the effect of:

• First, construct the query filtered down to specific tables
• If the query being constructed is <2048 characters (?), use it
• If the query is too long, just use show table extended like '*' instead

The relevant bit is here in the dbt-databricks adapter, where it's constructing a "schema relation" with identifier='<table_1>|<table_2>|...'.

Something like (naive pseudo-code):

identifier=("|".join(table_names) if len("|".join(table_names)) < 2000 else `"*"`),

So I think this wants to be an issue in that repo, instead of here in dbt-core. Do you mind reopening over there?