search

BurntSushi / xsv

A fast CSV command line toolkit written in Rust.
The Unlicense
10.23k stars 317 forks source link

--compress-program option for xsv split? #303

Open d33tah opened 2 years ago

d33tah commented 2 years ago

Today I learned how fast xsv split can be. The only thing that I feel that it's missing is a --compress-program switch (or similar) that would allow the user to compress created split files. @BurntSushi do you feel like it's a good fit for this tool?

BurntSushi commented 2 years ago

Nope. It seems totally orthogonal to me. You can always compress them after the split command.

d33tah commented 2 years ago

@BurntSushi Thanks for replying! The problem is that you need to store files in the uncompressed form in the meantime. Which, for huge CSV files - which benefit from splitting the most - can be quite cumbersome. In my case I decided to mount a compressed btrfs system to satisfy xsv's requirements, but having a way to compress data before saving to the disk would IMO solve the problem better.

LemmingAvalanche commented 1 year ago

@d33tah See qsv snappy.

d33tah commented 1 year ago

@LemmingAvalanche thanks!