search

gyscos / zstd-rs

A rust binding for the zstd compression library.
MIT License
527 stars 112 forks source link

Can we control the stack size of zstd? #225

Open ritchie46 opened 1 year ago

ritchie46 commented 1 year ago

Thanks maintaining this library. We use this downstream in polars and arrow2. Now we get a segfault triggered by a stack overflow during zstd decompression of large parquet files.

Is there a way for us to set a stack size parameters?

Stack trace can be found here; https://github.com/pola-rs/polars/issues/7237#issuecomment-1467777097

gyscos commented 1 year ago

Hi,

Not that I know of. This might be a question for https://github.com/facebook/zstd.

It does sound surprising though. I wouldn't expect the zstd lib to have a stack usage that grows that much with the size of the input.

(Just to be clear, you're not decoding into a stack-allocated array, are you?)