The above filters out all records not satisfying the test. However, it is often desirable to keep all the records, instead marking the records to indicate the matches. The following command does this, adding a hew field, IsPrimaryColor populated with values 1 or 0 to indicate pass or not.
Adding the label field is straightforward. In the main loop, instead of choosing to output a line or not, an indicator is appended. However, the additional conditional tests in the loop caused a performance degradation. This was partly due to the recent addition of the --count option, which counts the number of records satisfying the criteria. The performance degradation was minor for wide files with long lines, but substantial for narrow files.
To regain performance the code was templatized to reduce the number of tests in the main loop. In addition, some changes to BufferedOutputRange to streamline that code. It had also added some additional checks as part of the recent --line-buffered support. Between the two changes all the original performance was regained, and possibly a bit more.
Description
This PR adds a new feature to
tsv-filter: Marking each record as either passing the filter test, or not.Consider the following command, which identifies lines where the
Colorfield is a primary color.The above filters out all records not satisfying the test. However, it is often desirable to keep all the records, instead marking the records to indicate the matches. The following command does this, adding a hew field,
IsPrimaryColorpopulated with values1or0to indicate pass or not.The label values can be customized using the
--label-valuesoption. To change the above to usedtrueandfalse, run:Implementation
Adding the label field is straightforward. In the main loop, instead of choosing to output a line or not, an indicator is appended. However, the additional conditional tests in the loop caused a performance degradation. This was partly due to the recent addition of the
--countoption, which counts the number of records satisfying the criteria. The performance degradation was minor for wide files with long lines, but substantial for narrow files.To regain performance the code was templatized to reduce the number of tests in the main loop. In addition, some changes to
BufferedOutputRangeto streamline that code. It had also added some additional checks as part of the recent--line-bufferedsupport. Between the two changes all the original performance was regained, and possibly a bit more.