PRs adds exec_streamed method that returns Arrow Batches via stream while downloading/converting response. This reduces memory usage for large datasets as the records can be processed by chunks and improves performance by giving access for already loaded records.
I originally made RawQueryResult to always return result via stream but then realized that there is a polars dependency that requires RawQueryResult in bytes as it can't use async functionality to convert stream to bytes (defines TryFrom that is always sync)
With this change I was finally able to perform queries agains the very large snowflake_sample_data.tpch_sf100dataset.
PRs adds
exec_streamedmethod that returns Arrow Batches via stream while downloading/converting response. This reduces memory usage for large datasets as the records can be processed by chunks and improves performance by giving access for already loaded records.MAX_CHUNK_DOWNLOAD_WORKERS(10)download workers are used: https://github.com/snowflakedb/gosnowflake/blob/master/rows.go#L22RawQueryResultto always return result via stream but then realized that there is a polars dependency that requires RawQueryResult in bytes as it can't use async functionality to convert stream to bytes (defines TryFrom that is always sync)snowflake_sample_data.tpch_sf100dataset.