Closed pixie79 closed 1 year ago
Hi @pixie79 - couple of things:
dbt-core
and no longer dbt_utils
type_string()
macro in dbt-core
(see https://docs.getdbt.com/reference/dbt-jinja-functions/cross-database-macros#type_string) which doesn't have a spark dispatch. I don't think spark-utils has that either. If Glue really looks like spark to dbt, then the dbt-spark adapter would have to implement the proper data type macro, so that dbt-expectations gets the correct data type for strings.
Hope that makes sense! In the short term you could try implementing a version of type_string()
locally, something like
{% macro spark__type_string() -%}
STRING
{%- endmacro %}
@clausherther Thanks that temp fix does solve the issue. I will try and open an issue on dbt-core.
I have been trying this test using the DBT-Glue adapter which I believe uses the dbt-spark plugin underneath however it appears to be casting the list of columns as TEXT which is unsupported.
As far as I can tell that is generated via - dbt_utils.type_string(), i think it should be CASTing as STRING?
Is the a way to override this?