Growing unpacker's buffer by a set factor

[sztomi]

I would like to use an application-defined buffer growing strategy, basically (pseudo code):

if (data_size > unpacker.buffer_capacity()) {
    unpacker.resize(unpacker.size() * RESIZE_FACTOR);
}

However, there is no size() and resize(). AFAIK, reserve_buffer could be used to to add the required number of bytes, but I found no way to get the current reserved buffer size. buffer_capacity returns the remaining bytes (m_free), but m_used is not returned by anything. Is there a way to get this information from the unpacker without changing its interface? Or do I need to implement some sort of bookkeeping for the current buffer size?

[redboltz]

@sztomi , thank you for reporting the issue. Unfortunately, there is no way to do that, so far.

Here is unpacker's internal mechanism: http://www.slideshare.net/taka111/msgpackc

The unpacker's internal buffer expansion is defined as follows: https://github.com/msgpack/msgpack-c/blob/master/include/msgpack/v2/parse.hpp#L839

Currently, RESIZE_FACTOR is always 2 (hard coded).

I can introduce MSGPACK_UNPACK_EXPANSION_FACTOR macro or unpacker::set_expansion_factror(std::size_t facror).

However, providing resize() is not so easy. The buffer contains atomic counter, and some part of buffer is already consumed.

If you use x3 parse (experimental feature) instead of unpacker, you can manage the buffer by yourself. See https://github.com/msgpack/msgpack-c/blob/master/example/x3/stream_unpack.cpp#L141

[sztomi]

Thanks. How does the new parser perform in comparison to the old one?

[redboltz]

Thanks. How does the new parser perform in comparison to the old one?

The performance depends on Spirit.X3 parser. I wrote simple comparison program. It's not a stream unpacking.

The new parser is 2 times slower than older one, so far. I expect Spirit.X3 performance improvement in the future.

[redboltz]

This is the implementation of the new parser: https://github.com/msgpack/msgpack-c/blob/master/include/msgpack/v2/x3_parse.hpp

[sztomi]

Thanks, I'm looking forward to it.