pull vls chunked-filter work, see chunked-buffer, serde_bolt, txoo crates for details.
The tldr is that we now use the chunked_buffer::GenericChunkedBuffer structure, imported here as serde_bolt::NonContiguousOctets for both the MQTT bytes, and the ProofType::Filter structure in txoo.
Key features of NonContiguousBytes:
Max contiguous memory block size: 1024 bytes.
Bytes are progressively de-allocated from memory as they are read.
As bytes are written to the structure, no reallocation of the bytes themselves ever occurs. We instead keep allocating independent, 1024-byte blocks of memory as necessary.
The use of this structure achieves the following:
Completely eliminates the source of all of our out-of-memory crashes up until this point.
Reduces the max contiguous memory block size from 45KB to 1KiB when creating both of these structures, which increases the efficiency of our memory usage.
Decreases peak memory usage by progressively deallocating the mqtt bytes and allocating the ProofType::Filter bytes as the parse occurs. We previously needed space for 2 45KB contiguous memory blocks, (1 for the MQTT bytes, and one for the Filter bytes). This resulted in very premature out-of-memory crashes, ie memory crash when still 90KB of memory available. This is because the MQTT bytes were deallocated only once the full Filter structure had been created.
total: 91K, max block: 37K
msgs::read
memory allocation of 39235 bytes failed
pull vls chunked-filter work, see
chunked-buffer
,serde_bolt
,txoo
crates for details.The tldr is that we now use the
chunked_buffer::GenericChunkedBuffer
structure, imported here asserde_bolt::NonContiguousOctets
for both the MQTT bytes, and theProofType::Filter
structure in txoo.Key features of
NonContiguousBytes
:The use of this structure achieves the following:
ProofType::Filter
bytes as the parse occurs. We previously needed space for 2 45KB contiguous memory blocks, (1 for the MQTT bytes, and one for theFilter
bytes). This resulted in very premature out-of-memory crashes, ie memory crash when still 90KB of memory available. This is because the MQTT bytes were deallocated only once the fullFilter
structure had been created.