Fails deserializing multiple objects

[zacps]

Pickle is capable of storing multiple objects in the same file as each dump is self-contained.

Hence the following:

from pickle import dump, load
with open('test.gz', 'wb') as f:
    dump(1, f)
    dump(2, f)
with open('test.gz', 'rb') as f:
    print(load(f))
    print(load(f))

Returns

1
2

But with this library:

from compressed_pickle import dump, load
with open('test.gz', 'wb') as f:
    dump(1, f, compression='gzip')
    dump(2, f, compression='gzip')
with open('test.gz', 'rb') as f:
    print(load(f, compression='gzip'))
    print(load(f, compression='gzip'))

It returns

1
\compress_pickle\compress_pickle.py in load(path, compression, mode, fix_imports, encoding, errors, buffers, arcname, set_default_extension, unhandled_extensions, **kwargs)
    334     else:
    335         try:
--> 336             output = pickle.load(  # type: ignore
    337                 io_stream,
    338                 encoding=encoding,

EOFError: Ran out of input

[lucianopaz]

Oh, that would be nice to have. I think that some of the compression packages, like zip, will make this difficult to support. Anyway, thanks for reporting @zacps! I'll see what I can do about this functionality when I get some time.

[zacps]

With zip I think you could do it by writing each call as an individual file in the zip archive. It's possible to add files to a zip without rewriting anything other than the central directory.

My approach would be to check if the current object is seekable, and if it is and the cursor position is non-zero assume we're in the middle of an archive. At that point read the central directory to find where the start of the next file should be, write it, then write the central directory back in.

At load:

• If the cursor is at zero check if there are multiple files, if there isn't just read the first entry. Otherwise read the first entry and set the cursor position to the start of the next file.
• On next call read the file from the current cursor location, set the cursor to start of the next and so on
• Panic on EOF

[eode]

Maybe from an API standpoint, considering that some compression algorithms will support multiple objects and some won't, you could simply add an allow_multiple=True to be added wherever compression is specified (thus giving it a chance to reject an unsupported compression algorithm/allow_multiple pair).