search

BurntSushi / xsv

A fast CSV command line toolkit written in Rust.
The Unlicense
10.29k stars 317 forks source link

Line buffering / streaming data (as with `tail -f`) #279

Open AntonHermann opened 2 years ago

AntonHermann commented 2 years ago

While it's perfectly reasonable to require reading the whole input before processing it, as discussed in #192 for some other subcommands, like xsv fmt or xsv select I see no obvious reason why they shouldn't be line buffered, although that seems to be the case.

If there is some kind of technical limitation I'm overseeing, would a --streaming option or something alike be possible?

phiresky commented 2 years ago

I don't think it actually waits for the end of the input file, it just reads the input in chunks.

So tail -f x.csv | xsv... works, it just outputs stuff in 32kB chunks. You can see this when you use a file where data is rapidly appended to.

So it should be possible to build this without any large modifications by just reducing the buffer size, possibly here