Open tasket opened 2 years ago
data: string|bytes - input data block, length limited by 2Gb by Python API
The size limitation noted in the python-zstd Readme appears to be a side-effect of not supporting Python buffer protocol. The simple .compress()
methods for the built-in modules do not have the 2GB limitation. This means a coder may have to introduce file size guard conditions specific only to python-zstd.
Problem
The lack of compatibility with certain buffer types makes this module harder to use and less efficient; one cannot do a simple drop-in replacement of the built-in compression modules.
An in-memory conversion of
memoryview()
andmmap()
tobytes()
is first required before zstd can be used on the data; this introduces code complexity (making an exception for zstd) and dramatically increases memory consumption.(Note that converting a memoryview to read-only has no effect here.)
Solution
Support the same buffer types that the Python built-in compression libraries accept.