Currently dbt-athena allows for multiple threads for computation. This is causing errors when generating multiple boto3.client calls simultaneously as it is not thread safe. In order to rectify this issue, we apply a thread lock in the adapter to temporarily lock the threads to allow for safe creation of each boto3 client. This addresses the endpoint_resolver and credential_provider issues many of us have been seeing frequently.
Closes #41
pytest adapter
I also updated the pytest-dbt-adapter version as 0.5.1 failed all tests. Bumping the version to 0.6.0 appears to work appropriately with dbt-core 1.0.3.
Tomme
commented
2 years ago
endpoint_resolver & credential_provider
Currently dbt-athena allows for multiple threads for computation. This is causing errors when generating multiple
boto3.clientcalls simultaneously as it is not thread safe. In order to rectify this issue, we apply a thread lock in the adapter to temporarily lock the threads to allow for safe creation of each boto3 client. This addresses theendpoint_resolverandcredential_providerissues many of us have been seeing frequently.pytest adapter
I also updated the pytest-dbt-adapter version as 0.5.1 failed all tests. Bumping the version to 0.6.0 appears to work appropriately with dbt-core 1.0.3.