Filter Processor Benchmarking

[winder]

Summary

Add new benchmarks Benchmarks for filters.

The results suggest that we're doing something extremely inefficient when setting up each additional filter.

Applying one filter to 1 or 25000 transactions has an almost negligible difference. On the other hand there seems to be a fixed 8ms overhead for each filter.

The numbers in each report are as follows:

• number of transactions in the test block
• number of string equality filters
• number of regex filters
• number of numeric filters.

num txn	eq filters	regex filters	numeric filters
25000	1000	0	0
1	1000	0	0
25000	1	0	0
1	1	0	0
25000	0	1000	0
1	0	1000	0
25000	0	1	0
1	0	1	0
25000	0	0	1000
1	0	0	1000
25000	0	0	1
1	0	0	1

BenchmarkProcess2/txn_25000_equal_1000_regex_0_numeric_0
BenchmarkProcess2/txn_25000_equal_1000_regex_0_numeric_0-8                 1    7791642627 ns/op
BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0
BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0-8                     1    8057424687 ns/op
BenchmarkProcess2/txn_25000_equal_1_regex_0_numeric_0
BenchmarkProcess2/txn_25000_equal_1_regex_0_numeric_0-8                  136       8644760 ns/op
BenchmarkProcess2/txn_1_equal_1_regex_0_numeric_0
BenchmarkProcess2/txn_1_equal_1_regex_0_numeric_0-8                      133       8780231 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_1000_numeric_0
BenchmarkProcess2/txn_25000_equal_0_regex_1000_numeric_0-8                 1    2271617844 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_1000_numeric_0
BenchmarkProcess2/txn_1_equal_0_regex_1000_numeric_0-8                     1    2330539073 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_1_numeric_0
BenchmarkProcess2/txn_25000_equal_0_regex_1_numeric_0-8                  384       2741222 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_1_numeric_0
BenchmarkProcess2/txn_1_equal_0_regex_1_numeric_0-8                      414       2810811 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1000
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1000-8                 6     206244626 ns/op
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1
BenchmarkProcess2/txn_25000_equal_0_regex_0_numeric_1-8                 2234        489659 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000-8                     6     197892161 ns/op
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1
BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1-8                     2068        508157 ns/op

[codecov[bot]]

Codecov Report

Merging #146 (0e01345) into master (442791a) will increase coverage by 4.05%. Report is 55 commits behind head on master. The diff coverage is 80.81%.

@@            Coverage Diff             @@
##           master     #146      +/-   ##
==========================================
+ Coverage   67.66%   71.71%   +4.05%     
==========================================
  Files          32       37       +5     
  Lines        1976     2747     +771     
==========================================
+ Hits         1337     1970     +633     
- Misses        570      678     +108     
- Partials       69       99      +30     

Files Changed	Coverage Δ
conduit/data/block_export_data.go	100.00% <ø> (+92.30%)	:arrow_up:
conduit/metrics/metrics.go	100.00% <ø> (ø)
conduit/pipeline/metadata.go	69.11% <ø> (ø)
conduit/plugins/config.go	100.00% <ø> (ø)
...duit/plugins/exporters/filewriter/file_exporter.go	81.63% <ø> (-1.06%)	:arrow_down:
conduit/plugins/importers/algod/metrics.go	100.00% <ø> (ø)
...gins/processors/filterprocessor/fields/searcher.go	77.50% <ø> (ø)
...ins/processors/filterprocessor/filter_processor.go	83.82% <ø> (+3.54%)	:arrow_up:
...plugins/processors/filterprocessor/gen/generate.go	34.28% <ø> (ø)
conduit/plugins/processors/noop/noop_processor.go	64.70% <ø> (+6.81%)	:arrow_up:
... and 21 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[shiqizng]

how do you tell from the output that there's fixed 8ms overhead for each filter?

[tzaffi]

BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000 BenchmarkProcess2/txn_1_equal_0_regex_0_numeric_1000-8 6 197892161 ns/op

It seems that way because when running 1000 filters, it takes 8 seconds. EG:

BenchmarkProcess2/txn_0_equal_1000_regex_0_numeric_0-8                     1    8057424687 ns/op

[tzaffi]

197892161

But numeric filters seems to be faster.

[winder]

1 filter / 1 txn takes about 8ms 1 filter / 25000 txn takes about 8ms 1000 filters / 1 txn takes about 8 seconds 1000 filters / 25000 txn takes about 8 seconds

I think it follows that the number of txn seems to be nicely optimized, but there seems to be some per-filter overhead somewhere.

[winder]

197892161

But numeric filters seems to be faster.

Good point, I was definitely looking at the first few results more than the other filter types.