Before this PR, each session writes result set into stdout and then it is passed to Interactive Session API response.
However, in larger use cases, it can cause IllegalSessionStateException due to its large result set size.
This PR introduces a new credential parameter use_arrow (experimental, by default it's set to False)
When you set it to True, then dbt-glue writes result set into a feather file (an Arrow IPC file) and uploads to S3 bucket (tmp-results directory under DBT_S3_LOCATION). With this, each session passes only metadata (bucket and key location) without actual data. It reduces the amount of data passed over Interactive Session API.
This changes dbt-glue to call GetSession and GetSecurityConfiguration API to retrieve configured security configuration.
When SecurityConfiguration is set to the session, then dbt-glue refers it and follows its encryption mode to upload/download the feather file.
Description
This PR is to add a new credential parameter
use_arrow
as experimental feature to achieve better scalability. (This is the new version of https://github.com/aws-samples/dbt-glue/pull/313)Before this PR, each session writes result set into stdout and then it is passed to Interactive Session API response. However, in larger use cases, it can cause
IllegalSessionStateException
due to its large result set size.This PR introduces a new credential parameter
use_arrow
(experimental, by default it's set toFalse
)True
, then dbt-glue writes result set into a feather file (an Arrow IPC file) and uploads to S3 bucket (tmp-results
directory underDBT_S3_LOCATION
). With this, each session passes only metadata (bucket and key location) without actual data. It reduces the amount of data passed over Interactive Session API.GetSession
andGetSecurityConfiguration
API to retrieve configured security configuration.Checklist
CHANGELOG.md
and added information about my change to the "dbt-glue next" section.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.