fancycode / pylzma

Python bindings for the LZMA library
http://www.joachim-bauch.de/projects/pylzma/
GNU Lesser General Public License v2.1
233 stars 72 forks source link

Feature: Add support for streaming ArchiveFiles #70

Open Peilonrayz opened 5 years ago

Peilonrayz commented 5 years ago

I have some large files that I need to extract, given their size and that pylzma reads the entire file into memory, can a stream API be provided for py7zlib.ArchiveFile objects.

This could be via a read_streamed(chunk_size=...) -> Iterator[bytes] function or allowing a size to be entered into read - read(amount=None) -> bytes. Allowing users to roll their own read_streamed:

def read_streamed(archive, chunk_size):
    while True:
        data = archive.read(chunk_size)
        if not data:
            break
        yield data

This would allow:

BiatuAutMiahn commented 4 years ago

Any progress on this? Thank you

evilaliv3 commented 3 years ago

Also in the context of GlobaLeaks we would find this pretty useful.

Our need is to be able to perform zip+encryption for files bigger than the available server RAM.

For this reason we currently implement zip in streaming with a modified version of zipfile inspired form spideroa, but we are looking for the possibility to bind AES encryption to it