duckdb / dbt-duckdb

dbt (http://getdbt.com) adapter for DuckDB (http://duckdb.org)
Apache License 2.0
852 stars 77 forks source link

Glue registration not working "Format is not supported in Glue registrar." #132

Closed N-Clerkx closed 1 year ago

N-Clerkx commented 1 year ago

When running a single model aimed to upload a seed csv to a parquet file on S3 and registering it with AWS Glue the following error appears: "Format is not supported in Glue registrar."

Looking into the code it seems the file format is not passed properly into the "register_glue_table" function in impl.py.

example model:

{{
  config(
    materialized = 'external',
    format = 'parquet',
    location = 's3://example-bucket/example/data.parquet',
    glue_register = true
    )
}}
select 'dbt-duckdb rocks'

with profile:

example:
  outputs:
    dev:
      type: duckdb
      path: ":memory:"
      extensions:
        - httpfs
        - parquet
      settings:
        s3_region: 'eu-central-1'
        s3_access_key_id: "{{ env_var('AWS_ACCESS_KEY_ID') }}"
        s3_secret_access_key: "{{ env_var('AWS_SECRET_ACCESS_KEY') }}"
      format: 
  target: dev

I'd love to help out with submitting a fix, only not sure if this needs to be done in the macro or the python code. I'll try to create a PR to fix it