search

amazon-ion / ion-rust

Rust implementation of Amazon Ion
Apache License 2.0
147 stars 35 forks source link

Support stream reading/parsing #351

Open mlartz opened 2 years ago

mlartz commented 2 years ago

Given an Ion file of a series of records (e.g. an object in S3), we'd like to be able to stream read/parse this file, ultimately creating an async (Tokio) Stream of Ion records. Due to memory constraints (Lambda, Fargate, etc), we cannot read the entire file into memory and then iterate through the records.

zslayton commented 2 years ago

To clarify a bit: the Reader is already capable of reading a file without pulling the entire thing into memory. However, if the input source it is reading from contains incomplete data, the Reader would either block (e.g. waiting on a socket) or fail (e.g. at the end of a Vec<u8>).

We'd like to offer an API that allows the Reader to be resilient to such a use case. For example, calls to next() or read_string() might return an IonError::InsufficientData (or IonError::WouldBlock?), indicating that the user should try again once more data is available.

zslayton commented 1 year ago

We now have non-blocking text and binary readers, but they can't be constructed with the ReaderBuilder API yet. Once that's done (see #484), we can close this issue.