duckdb / dbt-duckdb

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
Apache License 2.0
797 stars 69 forks source link

Add option to fetch Gsheet data using range #233

Closed Thootje closed 11 months ago

Thootje commented 11 months ago

Related to: #213

Finally came around to make this PR to add the option to fetch Google Sheet data using a range instead of fetching the entire sheet.

It works by calling .get(range) on the sheet instead of .get_all_records() if a range is provided. Within the source .yml file, you have to add a range using the range key and, optional, headers under the headers key you want to use in case your selected range doesn't have any headers.

version: 2

sources:
  - name: test_table
    schema: output
    meta:
      plugin: gsheet
      key: some_id_here
      worksheet: data
      range: A1:B2
      headers:
        - column_a
        - column_b
    tables:
      - name: test_table

Finally if you don't have a matching amount of headers and number of columns in the selected range, it will give a nice exception. For example, the below source .yml fetches 3 columns, but only 2 headers are provided. This will be caught by the exception.

version: 2

sources:
  - name: test_table
    schema: output
    meta:
      plugin: gsheet
      key: some_id_here
      worksheet: data
      range: A1:C2
      headers:
        - column_a
        - column_b
    tables:
      - name: test_table
jwills commented 11 months ago

Thank you @Thootje!