eBay / tsv-utils

eBay's TSV Utilities: Command line tools for large, tabular data files. Filtering, statistics, sampling, joins and more.

https://ebay.github.io/tsv-utils/

Boost Software License 1.0

1.43k stars 80 forks source link

csv2tsv output buffer size tuning #305

Closed jondegenhardt closed 4 years ago

jondegenhardt commented 4 years ago

This PR changes the BufferedOutputRange buffer and flush sizes used by csv2tsv. The result is to flush data to standard output a bit quicker (smaller block writes) when csv2tsv is processing CSV files containing limited amounts of CSV escape syntax. This makes better use of memory and is slightly faster. Main benefit occurs in Unix pipelines. The next process in the pipeline receives data in smaller chunks increasing parallelism between pipeline stages.

codecov-commenter commented 4 years ago

Codecov Report

Merging #305 into master will not change coverage. The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #305   +/-   ##
=======================================
  Coverage   99.36%   99.36%           
=======================================
  Files          18       18           
  Lines        6943     6943           
=======================================
  Hits         6899     6899           
  Misses         44       44

Impacted Files	Coverage Δ
csv2tsv/src/tsv_utils/csv2tsv.d	`100.00% <100.00%> (ø)`