In # 319, We needed to revert multiple PRs including https://github.com/aws-samples/dbt-glue/pull/286.
This PR is to re-submit PR #301 to enable users to fix their glue_session_id name and re-use it.
Description
To re-use a glue session, we currently have to
1/ dbt run with glue_session_reuse: true and #glue_session_id commented
2/ copy paste the resulting glue session in glue_session_id and uncomment it.
If the session is stopped (you waited too long), step 2 will failed to re-use, you have to repeat the whole process.
Solution
We add the following logic, when a glue_session_id is provided by the user:
If the session does not exists or could not be fetched, it is created (with glue_session_id as id)
If the glue session exists, in READY state, we re-use it.
If the glue session exists, in TIMEOUT OR STOPPED state, we delete it and re-create it (with glue_session_id as id) - since there are no re-start in the glue session api
When a glue_session_id is not provided, the behavior of generating a unique glue_session_id is kept.
In # 319, We needed to revert multiple PRs including https://github.com/aws-samples/dbt-glue/pull/286. This PR is to re-submit PR #301 to enable users to fix their glue_session_id name and re-use it.
Description
To re-use a glue session, we currently have to 1/ dbt run with glue_session_reuse: true and #glue_session_id commented 2/ copy paste the resulting glue session in glue_session_id and uncomment it.
If the session is stopped (you waited too long), step 2 will failed to re-use, you have to repeat the whole process.
Solution
We add the following logic, when a glue_session_id is provided by the user:
When a glue_session_id is not provided, the behavior of generating a unique glue_session_id is kept.
Checklist
CHANGELOG.mdand added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.