Add zstandard compression feature

piskvorky / smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

MIT License

3.18k stars 383 forks source link

Add zstandard compression feature #801

Closed rlrs closed 7 months ago

rlrs commented 7 months ago

Adds zstandard compression support, as requested in #799.

I made it an optional feature, because it looks like every other feature that has dependencies is also an optional feature. Obviously this means that the tests only work if the feature is installed.

mpenkov commented 7 months ago

Whoops, looks like the linters failed. Can you please fix the error and push?

mpenkov commented 7 months ago

Never mind, fixed it myself. Thanks for taking care of this!

ddelange commented 6 months ago

Hi 👋

This PR only adds support for reading. @rlrs can you open another PR to fix writing?

In [1]: from smart_open import open

In [2]: with open('~/Downloads/test.zst', 'w') as fp:
   ...:     fp.write('abc')
   ...:
---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
Cell In[2], line 2
      1 with open('~/Downloads/test.zst', 'w') as fp:
----> 2     fp.write('abc')

UnsupportedOperation: not writable

mpenkov commented 6 months ago

Can you have a look at https://github.com/piskvorky/smart_open/pull/812 @ddelange?

If it looks good, we'll make another bugfix release.