Add use_arrow as experimental feature to achieve better scalability

moomindani commented 5 months ago

This PR is to add a new credential parameter use_arrow as experimental feature to achieve better scalability.

Before this PR, each session writes result set into stdout and then it is passed to Interactive Session API response. However, in larger use cases, it can cause IllegalSessionStateException due to its large result set size.
This PR introduces a new credential parameter use_arrow (experimental, by default it's set to False)
- When you set it to True, then dbt-glue writes result set into a feather file (an Arrow IPC file) and uploads to S3 bucket (tmp-results directory under DBT_S3_LOCATION). With this, each session passes only metadata (bucket and key location) without actual data. It reduces the amount of data passed over Interactive Session API.
- This changes dbt-glue to call GetSession and GetSecurityConfiguration API to retrieve configured security configuration.
- When SecurityConfiguration is set to the session, then dbt-glue refers it and follows its encryption mode to upload/download the feather file.

[x] I have signed the CLA
[x] I have run this code in development and it appears to resolve the stated issue
[x] This PR includes tests, or tests are not required/relevant for this PR
[ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-glue next" section.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

moomindani commented 5 months ago

Would you add a test case to run a dbt job with use_arrow feature enabled?

Added the test case.

moomindani commented 5 months ago

Closing this PR because I created another new one: https://github.com/aws-samples/dbt-glue/pull/323

aws-samples / dbt-glue