zstd compress OverflowError: size does not fit in an int

[sanjaysrikakulam]

Hi,

When running the following I get OverflowError: size does not fit in an int

Python version 3.7.7 zstd version: 1.4.8.1

import numpy as np
import zstd

data = np.arange(1000000000, dtype='int64').tobytes()
compressed_data = zstd.compress(data)

# Error:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: size does not fit in an int

My data will be a list of integers and sometimes may be of length more than 1 billion. Can you please comment or share any workaround?

Thank you!

[sergey-dryabzhinsky]

data: string|bytes - input data block, length limited by 2Gb by Python API

You should split your data in chunks by 2Gb size or less.

[sanjaysrikakulam]

Ah, I see, sorry.

Thank you! :-)