stephenberry / glaze

Extremely fast, in memory, JSON and interface library for modern C++
MIT License
973 stars 98 forks source link

glz::reader/glz::writer for incremental reading/writing #1019

Open stephenberry opened 1 month ago

stephenberry commented 1 month ago

Glaze doesn't easily support incremental reading and writing. There are times when we want to just parse the first element of an array, perform logic in our program, and then decide how what item to use to parse the rest of the array. This is common for formats that need header information.

Another use case is appending data to a buffer incrementally.

This issue proposes the addition of two stateful classes: gz::reader and glz::writer, which will enable incremental reading/writing.

Initially this will focus on array support, but object support will be considered in the design.

Example syntax:

glz::reader reader{buffer};
reader.read(glz::element<0>(value));
reader.read(glz::rest(value));
glz::writer writer{buffer};
writer.append(value);
write.append(value);
stephenberry commented 1 month ago

Addresses: #917

stephenberry commented 1 month ago

Note that partial reading, which is supported for JSON, handles this problem in another manner.

stephenberry commented 1 month ago

JSON pointer syntax reading also supports this in part. The main difference is that we want a core way to maintain iterators at the parsed location, for the best possible speed. I think this will also take a good step in the direction of handling streams of data.

stephenberry commented 1 month ago

The new read_allocated option in #1027 also provides a solution for partial reading.

stephenberry commented 1 month ago

read_allocated has been renamed to partial_read in #1028

ChristofferGreen commented 1 week ago

The new API design proposal looks nice.