boto session waiter is setup incorrectly causing session state change to wait until session timeout

hsirah commented 4 months ago

Describe the bug

Issue introduced in #323 .

The session_provisioning_timeout_in_seconds value is being passed in as delay. Because of this, every session state change waits until the timeout expires before proceeding to the next step.

Steps To Reproduce

Install v1.7.2 of dbt-glue
Run any dbt command. For example, dbt debug. Run with --debug to see detailed logs.
Execution will pause at session creation step waiting for session state to change to READY
- The next step will not be executed until session_provisioning_timeout_in_seconds expires and the block for WaiterError is triggered.
- The same behavior can be seen when session is being closed

Expected behavior

The wait function should check waiter model every delay seconds instead of every timeout seconds.

Screenshots and log output

(see timestamp for delay. session_provisioning_timeout_in_seconds using default value of 300 seconds)

19:32:28  Glue adapter: Session recreation initiated for dbt-glue__no-node, waiting it to be READY, currently in PROVISIONING state
19:37:29  Glue adapter: GlueConnection _init_session called for session_id : dbt-glue__no-node

System information

The output of dbt --version:

Core:
  - installed: 1.7.7
  - latest:    1.7.9 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - glue:  1.7.2 - Up to date!
  - spark: 1.7.1 - Up to date!

The operating system you're using: MacOS

The output of python --version: