search

dbt-labs / dbt-external-tables

dbt macros to stage external sources
https://hub.getdbt.com/dbt-labs/dbt_external_tables/latest/
Apache License 2.0
286 stars 115 forks source link

Infer schema for BigQuery .parquet files #270

Open LinasData opened 3 months ago

LinasData commented 3 months ago

Describe the feature

Currently it is mandatory to add data_type property when defining external tables columns. It would be nice for BigQuery to infer all columns as strings by default.

Describe alternatives you've considered

Tried this but it did not work. With CSV files works fine.

      # alternatively, BigQuery can infer your schema (columns + partitions)
      - name: event_inferred
        external:
          location: 'gs://bucket/path/*'
          options:
            format: parquet
            hive_partition_uri_prefix: 'gs://bucket/path/'

Who will this benefit?

Analytics engineers who work with Google Cloud services majority of the time.