Closed MichaelChirico closed 5 years ago
In benchmarks we are forcing evaluation so the whole benefits of laziness are being lost. Therefore I don't think we should focus on it, unless it is fast also when not lazy. It could be useful when we would for example groupby and subset, then subset can use laziness and read only required rows for groupby. We don't have such test at the moment.
I think your second point gets more at what I had in mind, but I can't seem to articulate what exactly I have in mind, so closing for now.
Seeing all sorts of praise for
vroom
as being "faster" thanfread
recently.It seems nobody is reading the small print --
vroom
is just a lazy reader, which means that while I/O itself will feel fast, subsequent analysis will be (much?) slower... not to mention I presume it rates a 0 on robustness to non-cookie-cutter csvs.It would be helpful to evaluate my suspicion quantitatively by building
vroom
into one of the pipelines here, not to mention it would be useful to include the start-up I/O costs of the other languages as well.