balena-os / balena-engine

Moby-based Container Engine for Embedded, IoT, and Edge uses
https://www.balena.io
Apache License 2.0
692 stars 66 forks source link

Avoid unbounded memory allocations when generating deltas #426

Closed lmbarros closed 1 year ago

lmbarros commented 1 year ago

This commit just updates librsync-go to v0.7.1, circbuf to v0.1.1 -- the real fix is on librsync-go. For reference, here's a copy of commit message of the librsync-go commit that interest us:

Before this commit, the delta output buffer (for LITERAL commands) was unbounded, so very long sequences on the new file without a match on the old file could allocate ginormous amounts of memory.

This commit adds a limit to this buffer, so that we impose a limit on the amount of memory we allocate at once. As a minor side effect, this also limits the size of the literal sequence on a single LITERAL command to the size of this output buffer.

In addition to limiting the buffer size, this commit also pre-allocates the buffer memory. This avoids frequent memory reallocations and copies, and generates less garbage to the garbage collector.

- What I did

- How I did it

- How to verify it

- Description for the changelog