dremio / dbt-dremio

dbt (data build tool) adapter for the Dremio
Apache License 2.0
44 stars 21 forks source link

[Bug]: ilike not replaced in generate docs command #198

Open donatobarone opened 1 year ago

donatobarone commented 1 year ago

Is there an existing issue for this?

Current Behavior

With a recent enhancement the TABLES statements have been modified to use = rather than ilike if the "dremio:exact_search_enabled": true is specified. https://github.com/dremio/dbt-dremio/issues/134 I am not sure if it was intended behaviour or not that is why marking it as a bug the dbt docs generate seems to be ignoring that and still fires requests with the ilike command.

/* {"app": "dbt", "dbt_version": "1.5.2", "profile_name": "data_product_demo", "target_name": "dev", "connection_name": "location"} */
with cte as (

      select (case when position('.' in columns.table_schema) > 0
            then substring(columns.table_schema, 1, position('.' in columns.table_schema) - 1)
            else columns.table_schema
        end) as table_database
        ,(case when position('.' in columns.table_schema) > 0
            then substring(columns.table_schema, position('.' in columns.table_schema) + 1)
            else 'no_schema'
        end) as table_schema
        ,columns.table_name
        ,lower(t.table_type) as table_type
        ,cast(null as varchar) as table_comment
        ,column_name
        ,ordinal_position as column_index
        ,lower(data_type) as column_type
        ,cast(null as varchar) as column_comment
        ,cast(null as varchar) as table_owner
      from information_schema."tables" as t
      join information_schema.columns
        on (t.table_schema = columns.table_schema
        and t.table_name = columns.table_name)
      where t.table_type <> 'SYSTEM_TABLE'
      and (ilike( t.table_schema, 'tablename'))
    )

    select *
    from cte
    order by table_schema
      ,table_name
      ,column_index

Expected Behavior

/* {"app": "dbt", "dbt_version": "1.5.2", "profile_name": "data_product_demo", "target_name": "dev", "connection_name": "location"} */
with cte as (

      select (case when position('.' in columns.table_schema) > 0
            then substring(columns.table_schema, 1, position('.' in columns.table_schema) - 1)
            else columns.table_schema
        end) as table_database
        ,(case when position('.' in columns.table_schema) > 0
            then substring(columns.table_schema, position('.' in columns.table_schema) + 1)
            else 'no_schema'
        end) as table_schema
        ,columns.table_name
        ,lower(t.table_type) as table_type
        ,cast(null as varchar) as table_comment
        ,column_name
        ,ordinal_position as column_index
        ,lower(data_type) as column_type
        ,cast(null as varchar) as column_comment
        ,cast(null as varchar) as table_owner
      from information_schema."tables" as t
      join information_schema.columns
        on (t.table_schema = columns.table_schema
        and t.table_name = columns.table_name)
      where t.table_type <> 'SYSTEM_TABLE'
      and  t.table_schem='tablename'
    )

    select *
    from cte
    order by table_schema
      ,table_name
      ,column_index

Steps To Reproduce

Just running dbt docs generate should do

Environment

- OS: Monterey 12.5.1
- dbt-dremio: 1.5.0
- Dremio Software: 23.1.5
- Dremio Cloud: N/A

Relevant log output

No response

donatobarone commented 1 year ago

Any update?

fabrice-etanchaud commented 1 year ago

the dbt docs may be using the get_catalog macro :

https://github.com/dremio/dbt-dremio/blob/8143c81a4e357ac90d1539192a8a6ee047d7e555/dbt/include/dremio/macros/adapters/metadata.sql#L15C10-L15C29

which has to be adapted too.

donatobarone commented 1 year ago

There are a few places where ilike seems to be used, so I guess for the nature of the optimisation it should be changed in all of those, or maybe creating a central function that wraps the logic and gets called in all the other macros.