search

lloyd / yajl

A fast streaming JSON parsing library in C.
http://lloyd.github.com/yajl
ISC License
2.15k stars 435 forks source link

streaming parsing for very long strings: new feature proposal #178

Open mewalig opened 8 years ago

mewalig commented 8 years ago

In the spirit of streaming, I'd like to propose an alternate approach for handling string callbacks, which would do so in a streaming fashion so that arbitrarily large strings can be handled with a fixed amount of memory and in a way that enables further streamed processing of strings. If this is already possible please let me know! as far as I can tell it currently is not.

So for example, there could be an additional string callback e.g.:

static int yajl_handle_string_partial(void *ctx, const unsigned char *stringVal, size_t len);

where yajl_handle_string_partial() would only be used if the string exceeds the length of the buffer (without requiring the buffer to grow). The user can then choose to process the string however they want.

To take it a little further, options could provide flexibility for user-defined buffers e.g.

char my_buffer[4096];
yajl_config(handle, yajl_string_buffer, my_buffer);
yajl_config(handle, yajl_string_buffer_size, sizeof(my_buffer));

Any interest in adding this feature? Happy to help if so

nkhorman commented 8 years ago

The changes that I've made to add a generic framework for sourcing the json content might be what your looking for. https://github.com/lloyd/yajl/pull/160 / https://github.com/nkhorman/yajl