expect_column_values_to_be_of_type

[pixie79]

I have been trying this test using the DBT-Glue adapter which I believe uses the dbt-spark plugin underneath however it appears to be casting the list of columns as TEXT which is unsupported.

11:42:15        from (
11:42:15          with relation_columns as (
11:42:15    
11:42:15            
11:42:15            select
11:42:15                cast('GL_HEADER_KEY' as TEXT) as relation_column,
11:42:15    ------------------------------------^^^
11:42:15                cast('STRING' as TEXT) as relation_column_type
11:42:15            union all

As far as I can tell that is generated via - dbt_utils.type_string(), i think it should be CASTing as STRING?

Is the a way to override this?

[clausherther]

Hi @pixie79 - couple of things:

• FYI, we only support and test dbt-expectations on Postgres, BigQuery and Snowflake, so I won't be able to diagnose this much further
• Looks like you might be running an older version of dbt-expectations, b/c as of 0.6.0 all data type macros (except timestamp for now) come from dbt-core and no longer dbt_utils
• Regardless, this is an issue with the type_string() macro in dbt-core (see https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros#type_string) which doesn't have a spark dispatch. I don't think spark-utils has that either. If Glue really looks like spark to dbt, then the dbt-spark adapter would have to implement the proper data type macro, so that dbt-expectations gets the correct data type for strings. Hope that makes sense!

[clausherther]

In the short term you could try implementing a version of type_string() locally, something like

{% macro spark__type_string() -%}
    STRING
{%- endmacro %}

[pixie79]

@clausherther Thanks that temp fix does solve the issue. I will try and open an issue on dbt-core.