Feature/glue session per model - Githubissues

aws-samples / dbt-glue

This repository contains the dbt-glue adapter

Apache License 2.0

102 stars 69 forks source link

Feature/glue session per model #234

Closed nandubatchu closed 1 year ago

nandubatchu commented 1 year ago

resolves https://getdbt.slack.com/archives/C02R4HSMBAT/p1693226615486969

Description

tried to bring an optional feature to enable individual glue sessions (with possible overrides of worker configurations) per model
can be enabled using the enable_session_per_model: true flag in the profiles.yml

can override the session configurations by putting the corresponding config under meta config for a model

{{
config(
    meta={
        "workers": 3,
        "idle_timeout": 10,
        "worker_type": "G.2X"
    }
)
}}

Checklist

[ ] I have signed the CLA
[X] I have run this code in development and it appears to resolve the stated issue
[ ] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

nandubatchu commented 1 year ago

@menuetb pls review and merge