Closed OElesin closed 1 year ago
Hi, Thanks for let us know. We are working on implementing the snapshot feature, but it's not yet available, I'll let you know once it will be available.
@menuetb , any updates on this?
Hi, I just developed a beta which works for snapshots using Delta.
You can install this beta using the following command: python3 -m pip install --index-url https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ dbt-glue
You can find the documentation for the configuration here: https://github.com/aws-samples/dbt-glue/blob/delta_lake_integration/README.md#delta
You need to install delta glue connectors to make it works, here is the doc for that: https://docs.aws.amazon.com/glue/latest/ug/connectors-chapter.html
Here is an example that works on my account:
{% snapshot demosnapshot %}
{{
config(
strategy='timestamp',
target_schema='dbt_demo_01',
unique_key='customer_id',
updated_at='dt',
file_format='delta'
)
}}
select * from {{ ref('customer_delta') }}
{% endsnapshot %}
I just published a new version of the adapter supporting snapshots for both delta and hudi, this feature is now available
Describe the bug
When creating snapshots, I get the error that the target table for the snapshots to be loaded into does not exist. This causes a failure in the
dbt snapshot
execution.Steps To Reproduce
Add IAM Role with Lake Formation permissions used by Glue Interactive Sessions
$ dbt snapshot
in the terminalExpected behavior
According to the dbt example on snapshots, no error is expected
Screenshots and log output
dbt.log output below
System information
The output of
dbt --version
:The operating system you're using: MacOS
The output of
python --version
: python3Additional context
Definitely, the snapshot table would not exist and permissions have been given to the Glue Interactive Sessions IAM role to create tables.