search

eerimoq / detools

Binary delta encoding tools.
Other
157 stars 16 forks source link

Memory leak while applying patches #7

Closed tspivey closed 1 year ago

tspivey commented 1 year ago

There seems to be a memory leak while applying patches. Tested on 0.51.0 from PyPI and latest master (3b0a0a7ccd12355483a2a3efabf95e4dafe0d49a).

Test script:

from io import BytesIO
import detools

l = 50000000
data1 = b'a' * l
data2 = b'b' * l

patch = BytesIO()
detools.create_patch(BytesIO(data1), BytesIO(data2), patch, compression='zstd', patch_type='hdiffpatch', algorithm='hdiffpatch', use_mmap=False)
for i in range(50):
    new = BytesIO()
    print(i)
    patch.seek(0)
    detools.apply_patch(BytesIO(data1), patch, new)

On my system, this uses over 2 GB of memory once it completes (with python -i).

Adding free(temp_cache_p ); to the end of m_apply_patch in hdiffpatch.cpp reduces the usage to under 200 MB. Is this the correct solution?

eerimoq commented 1 year ago

Yes, that looks like a bug and the free you suggest seems like a proper fix. Feel free to create a PR with the fix and I'll review and publish a new release once fixed.