lquerel / gcp-bigquery-client

GCP BigQuery Client (Rust)
Apache License 2.0
92 stars 60 forks source link

feat(job): Make query pagination an async stream #24

Closed LiHRaM closed 1 year ago

LiHRaM commented 2 years ago

I've thought a bit about how we can make the pagination more flexible for diverse usecases, and I'm sure people want to handle possible errors differently, so I suggest we go with something á la the following to make that possible:

The pagination becomes an asynchronous iterator over Result<Vec<TableRow>, BQError>. People are then able to stream through very large results without exhausting memory, and can use collect and flatten to imitate the previous implementation.

lquerel commented 1 year ago

Agree. Thanks for this improvement. I will publish soon a new release with this update.