dbt-labs / dbt-spark

dbt-spark contains all of the code enabling dbt to work with Apache Spark and Databricks
https://getdbt.com
Apache License 2.0
395 stars 221 forks source link

[ADAP-1012] Support for new agate data type in Spark #935

Closed gshank closed 10 months ago

gshank commented 10 months ago

dbt Core changes

To support integers in the "show" command, a new agate data_type, Integer, was added in dbt Core in core/dbt/clients/agate_helper.py. The agate table creation code is also used when creating empty tables for seeds, such as when a seed file has headers but doesn't have any rows. Because the "convert_agate_type" method did not include the new Integer type, "None" was returned and the sql generated by the macro "create_csv_table" was incorrect.

dbt Core pull request https://github.com/dbt-labs/dbt-core/pull/9004 adds a new "convert_integer_type" method and adds Integer to the list of agate types in "convert_agate_type".

Adapter changes

This pull request will pull in the new empty seed test and provide a "convert_integer_type" method.