unclear when indefinite length could be bad

[aep]

there's some subtle hints about indefinite length items not being desirable in some circumstances, but it's very unclear why. Actually i'm unsure why i would ever use definite lengths at all.

Using indefinite-length encoding allows
an encoder to not need to marshal all the data for counting, but it
requires a decoder to allocate increasing amounts of memory while
waiting for the end of the item.

• for unconstrained devices: why even bother. server memory is basically infinite. json works this way as well.
• for constrained devices: knowing the array is too large earlier in the byte-stream doesn't give you more options to deal with the problem. It's still too large. Also the parsed tree's size is vaguely related to the byte-streams size, so the problem already hit you before parsing the cbor.

someone also complained on reddit (cant find the link anymore) that indefinite length items are a design mistake in cbor, but i can't figure out why. They seem fine.

[cabo]

Hi Arvid,

let's first distinguish indefinite length strings (which are a construct build on top of definite length strings) from indefinite length containers (arrays, maps). I think you are talking about the latter.

There are two reasons why definite length containers may be the better choice:

• For containers with less than 24 elements/entries, definite length is one byte shorter. You may not care about that much, but for data that use large amounts of small arrays, that can make a difference.
• The decoder knows about the size (number of elements/entries) right from the start. That may be an indication that the container is too large, but the more interesting case is when that early size information allows you to decode data that is in the container in a more efficient way (e.g., when you can allocate the right application data structure at the right size).

If these are not important to you, be our collective guest and go indefinite as much as you want -- that's why that is in there. (Unless you need deterministic encoding, which is biased towards definite, but of course you can define your own deterministic encoding rules.)

[aep]

thanks,

maybe found the answer in https://github.com/msgpack/msgpack/issues/128#issuecomment-21143606 That is some languages require realloc on array append, which might copy, so that's alot slower than a known size alloc.

curious why protobuf opted for not sizing arrays at all tho