Open itayfisz opened 1 year ago
IronCompress is already on latest snappy version, but there is a bug in narrowing data type as you have mentioned. Thanks for reporting this, I'll try to reproduce and get some fixes in.
In the meantime, you can try to write in batches (row groups) as it looks like columns are massive anyway and readers will have issues decompressing them if ram is an issue.
Library Version
4.16.2
OS
Linux
OS Architecture
64 bit
How to reproduce?
It's hard to reproduce - It happens rarely when I try to write column with many rows, with very long string values. After calling WriteColumnAsync, I get
ArgumentOutOfRangeException
- see full error below. Seems that in IronCompress\Iron.cs, the returned output variable "len" is negative, which is then used to initialize an array, and as a result causing theArgumentOutOfRangeException
.This is probably because the
snappy::RawCompress
, called in api.cpp, has an integer overflow. Here's a bug on Snappy about it, which has already been fixed.I assume the solution would be to upgrade the Snappy version, but I'm not sure if it would just return a more informative error.
Failing test
No response