databricks / databricks-sql-go

Golang database/sql driver for Databricks SQL.
Apache License 2.0
37 stars 41 forks source link

PECO-1054 Expose Arrow batches to users #160

Closed rcypher-databricks closed 1 year ago

rcypher-databricks commented 1 year ago

Step one of exposing arrow batches directly to users. Moved the logic for iterating over the pages in a result set into ResultPageIterator.
Rows now composes in ResultPageIterator. Introduced Delimeter type. Delimeter tracks a start/end point and provides functions for determining if a point is within the delimiter range and the direction of the point if it is outside the delimeter range. Updated sparkArrowBatch, arrowRowScanner, columnRows, rows to use Delimiter. Updated the Fetch logic for cloudURL and localBatch so that the concurrentFetcher doesn't need to hold or pass through a Config instance.