Streaming encoding on the fly

[escherstair]

I don't know if this is possible or not. I have a fixed size array of data float data[100]; and I need to encode it in CBOR and send the encoded CBOR. I know how to use QCBOR to generate a large encoded UsefulBuf (in memory) and then iterate over it to send byte after byte.

Is there a way to use QCBOR to encode the array on the fly and to send out the data without having the large encoded UsefulBuf in memory?

For other CBOR library I see streaming.

[laurencelundblade]

No, it's not possible with the current interface.

I can imagine adding a streaming mode at some point. It would have to work only with indefinite length maps and arrays. Some sort of flush API would be added.

I will leave this open and mark it as a feature enhancement to add some day. It won't be anytime soon as the list is pretty long.

[Anrock]

It would have to work only with indefinite length maps and arrays.

Um, not necessary. I suppose the difficulty of encoding sized map/arrays is that size bytes are encoded before array/map items and could vary in size themselves so you have to keep that whole array in memory before flushing in order to fill introduction bytes and possibly shift data if element count is big.

But you can workaround that by providing a function that opens sized array but requires array size argument. Then it's programmers responsibility to provide exactly N elements before calling array close function. You also should be able to track that cheaply with counter variable for error detecting.

[laurencelundblade]

Yes, you are right. And doing that will require an API change for QCBOR arrays and maps. Thx for correcting that.

[escherstair]

But you can workaround that by providing a function that opens sized array but requires array size argument. Then it's programmers responsibility to provide exactly N elements before calling array close function.

Yes, this is the approach I reccomend. The other tricky point is that if this kind of array is included in a map, the header of the map must know how many element it contains, when the map itself is open (because it mujst contain the right values when it's flushed).

I've been able to create a simplified scenario of streaming array with QCBOR. It works as I need, but I'm not satisfied aboiut the API (not very portable/expandable). I create void QCBOREncode_AddSpaceForStreamingArrayItemInMap(QCBOREncodeContext *pMe); that tweaks pMe->nesting.pCurrentNesting->uCount Then void QCBOREncode_OpenStreamingArrayInMap(QCBOREncodeContext *pMe, const char *szLabel, size_t array_size); that opens the array, sets pMe->nesting.pCurrentNesting->uCount as array_size and closes the array. And, finally, a function to flush chunks of elements of the array.

But, as I wrote, this works in my project but the API is quite ugly.