aws-samples / dbt-glue

This repository contains de dbt-glue adapter
Apache License 2.0
91 stars 59 forks source link

File Format JSON - Table fails to register #113

Open pixie79 opened 1 year ago

pixie79 commented 1 year ago

Describe the bug

When you set the file_format to JSON and complete a DBT run, the JSON files get written to the correct path on S3 however, the glue data catalogue fails to get updated with the new table.

Steps To Reproduce

Set the model properties as follows:

{{ config(
    partition_by=['year','month','day','hour'],
    file_format='json',
    materialized='table',
    custom_location='s3://' + var('AWS_ACCOUNT_ID') + '-' + var('AWS_REGION') + '-partner-data/egress/posting_event'
) }}

Expected behavior

Like when the file_format is parquet or hudi I would expect the process to complete without error and register my new table in the glue data catalogue.

Screenshots and log output

Error returned to the console is: IllegalArgumentException: Can not create a Path from an empty string

System information

The output of dbt --version:

Core:
  - installed: 1.3.0
  - latest:    1.3.0 - Up to date!

Plugins:
  - redshift: 1.3.0 - Up to date!
  - postgres: 1.3.0 - Up to date!
  - spark:    1.3.0 - Up to date!

The operating system you're using:

The output of python --version:

Python 3.10.8

Additional context

Add any other context about the problem here.

pixie79 commented 1 year ago

I can confirm #114 does not solve this issue, but the error appears to be similar.