dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
297 stars 119 forks source link

Leverage snowflake infer_schema feature to auto-generate external tables schema #139

Closed alejobs closed 1 year ago

alejobs commented 2 years ago

Describe the feature

Currently we need to manually define all columns and data types coming from the external files. With this new feature we could auto-generate external tables schema by leveraging the infer_schema or generate_column_description in Snowflake: https://docs.snowflake.com/en/sql-reference/functions/generate_column_description.html

JoseRFJuniorLLMs commented 1 year ago

-- FOR INFERRING THE SCHEMA IN THE STAGE FILES select ','||column_name||' '||type|| '\n' from table ( infer_schema (location=>'@DEV.RAW_T2.USERS' , file_format=> '"DEV"."DEV_SCHEMA".PARQUET')) order by column_name;