duckdb / dbt-duckdb

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
Apache License 2.0
881 stars 78 forks source link

Update Excel plugin with output support #260

Closed JCotton1123 closed 11 months ago

JCotton1123 commented 1 year ago

Example profiles.yaml:

     plugins:
        - module: excel
          alias: accounts_file
          config:
            output:
              engine: 'xlsxwriter'
              engine_kwargs:
                # See https://xlsxwriter.readthedocs.io/workbook.html#Workbook
                options:
                  # See https://xlsxwriter.readthedocs.io/working_with_memory.html#memory-perf
                  strings_to_numbers: true
                  remove_timezone: true
                  use_zip64: true
                  nan_inf_to_errors: true
              file: "files/accounts.xlsx"
              index: false  # Don't output row number
              header_styling: false  # Disable default Pandas header styling

Example model:

{{ config(materialized='external', plugin='accounts_file', overrides={'sheet_name': 'New Accounts'}) }}

SELECT *
FROM {{ ref('accounts') }}
WHERE status = 'NEW'
jwills commented 1 year ago

(don't worry about the MotherDuck test failure, that's expected until they upgrade to 0.9.0)

JCotton1123 commented 1 year ago

@jwills the one thing this PR is missing are tests. If you can provide some guidance there I am happy to implement some. Also, thanks again for taking the time to chat this morning.

jwills commented 1 year ago

@JCotton1123 hey sorry for the lag here, I'm at a conference this week. Re: testing, I think the best move would be to extract the Excel-related bits of https://github.com/duckdb/dbt-duckdb/blob/master/tests/functional/plugins/test_plugins.py into a new test_excel.py file in that same directory which would add support for exercising the output-side of the plugin as well as the read-side that is exercised there now.

Take a look at the flow there and see if you're up for it; dbt functional tests are a little squirrely to understand, so if it's not clicking just let me know and I can add them in when I'm back around next week.

JCotton1123 commented 11 months ago

@jwills just getting back to this...I attempted to follow your suggestion for implementing a test. LMK if this needs any additional changes?

jwills commented 11 months ago

@JCotton1123 this looks great, thank you so much! The mypy linting error looks legit worth fixing and then I'm not totally sure what's up with the plugin test (maybe another dep needs to migrate from test_plugins.py to test_excel.py?) I'm happy to take a crack at it if you're busy otherwise, you just need to allow commits from upstream committers.

JCotton1123 commented 11 months ago

@jwills I believe those issues should be resolved 🤞

jwills commented 11 months ago

@JCotton1123! Yay-- thanks again!