Encode_into might not write data at the intended offset location if buffer is too small.

[MrPigss]

Description

When encoding into a buffer at an offset that is larger than the buffer size, the the msgpack Encoder.encode_into() function will start writing at offset = -1instead.

Example:

from msgspec.msgpack import Encoder

enc = Encoder()
buffer = bytearray(2) # bytearray of size 2

enc.encode_into("some serializable object", buffer, 4) # offset of size 4

print(buffer) # |> bytearray(b'\x00\x00\xb8some serializable object')

Notice it started at offset 2 instead of offset 4.

Expected behaviour:

encode_into starts writing at the requested offset, no matter how large the buffer currently is.

Note

Just mentioning this in the docs as intended behaviour is fine for me as longs as it's documented somewhere.

[jcrist]

This is the currently intended (or at least not unintended) behavior. In my mind there are 3 main use cases for encode_into, none of which would run into this issue:

1. Encoding into a pre-allocated buffer. In this case offset is never needed.

buffer = bytearray(1024)
for msg in msgs:
    enc.encode_into(msg, buffer)
    socket.sendall(msg)

2. Encoding into a pre-allocated buffer, leaving space for a prefix. This is the length-prefix framing example in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.

buffer = bytearray(64)
encoder.encode_into(msg, buffer, 4)  # offset for prefix, but 4 is less than 64 and both are hardcoded sizes (not data dependent)
n = len(buffer) - 4
buffer[:4] = n.to_bytes(4, "big")
socket.sendall(buffer)

3. Accumulating several writes into the same buffer, perhaps with a delimiter. In this case you're always writing to the end of the buffer, so offset is -1

```python
buffer = bytearray()
for msg in msgs:
    encoder.encode_into(msg, buffer, -1)
    buffer.append("\n")  # perhaps we're writing line-delimited json

---

That all said, I can see how the current behavior would be surprising, and would be open to changing it to automatically grow the buffer to support the specified offset.

Out of curiousity, what was your use case that led to finding this issue?

[MrPigss]

The use case was actually case 2.

Encoding into a pre-allocated buffer, leaving space for a prefix. This is the length-prefix framing example in the docs. In this case I kind of expected the pre-allocated buffer to always be longer than the known prefix length.

I just forgot to add the size on one of my buffers. Python has no problem doing x = bytearray() without size, and decode_into has no issue writing to this buffer. Just the right amount of coincidences happend and my encoded message could still be decoded (at first). It's only when I started to expand the tests that i suddenly got weird results, and messages that couldn't be decoded.

Since both just work, it was quite hard to pinpoint where my message got "corrupted". A mention about this in the docs would've saved me some time here.

For context i'm writing an ASGI RPC app (and client) using msgspec for serialisation.

[jcrist]

Hmmm, in that case maybe we should error instead if offset > len(buffer)? That would have caught your issue and pointed you to a better usage pattern. On the other hand, making it work by expanding the buffer is also easy, and would have negligible performance impact (at least for reasonable offsets) since we'd have to grow the buffer anyway to write the message. Waffling a bit here on what the best UX is.

[MrPigss]

I can see benefits for both options. Expanding the buffer will "Just work" and I can't really see a use case where you might not want that to happen, especially since encode_into expands anyway when necessary.

On the other hand going by the zen of python:

Explicit is better than implicit.

and

Special cases aren't special enough to break the rules

If anyone ever has a use case where they don't want this expanding to happen, it might be hard to figure out what is going on.