Closed tzaffi closed 1 year ago
Merging #128 (9918073) into master (442791a) will increase coverage by
4.32%
. Report is 52 commits behind head on master. The diff coverage is81.89%
.
@@ Coverage Diff @@
## master #128 +/- ##
==========================================
+ Coverage 67.66% 71.98% +4.32%
==========================================
Files 32 36 +4
Lines 1976 2695 +719
==========================================
+ Hits 1337 1940 +603
- Misses 570 657 +87
- Partials 69 98 +29
Files Changed | Coverage Δ | |
---|---|---|
conduit/data/block_export_data.go | 100.00% <ø> (+92.30%) |
:arrow_up: |
conduit/metrics/metrics.go | 100.00% <ø> (ø) |
|
conduit/pipeline/metadata.go | 69.11% <ø> (ø) |
|
conduit/plugins/config.go | 100.00% <ø> (ø) |
|
...duit/plugins/exporters/filewriter/file_exporter.go | 81.63% <ø> (-1.06%) |
:arrow_down: |
conduit/plugins/importers/algod/metrics.go | 100.00% <ø> (ø) |
|
...gins/processors/filterprocessor/fields/searcher.go | 77.50% <ø> (ø) |
|
...ins/processors/filterprocessor/filter_processor.go | 83.82% <ø> (+3.54%) |
:arrow_up: |
...plugins/processors/filterprocessor/gen/generate.go | 34.28% <ø> (ø) |
|
conduit/plugins/processors/noop/noop_processor.go | 64.70% <ø> (+6.81%) |
:arrow_up: |
... and 20 more |
:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more
If it's easy for you to run, it might be interesting to see Local test network 5 minute sprint
using different log levels to show how logrus/logging in general is impacting processing times.
If it's easy for you to run, it might be interesting to see
Local test network 5 minute sprint
using different log levels to show how logrus/logging in general is impacting processing times.
Can we add this as a task for #131 ?
Description
Allowing for moderate concurrency in the pipeline but without sacrificing its sequential integrity.
Summary of Changes
conduit/pipeline/common.go
: introducing generic retrying of pipeline methods viaRetries()
andRetriesNoOutput()
conduit/pipeline/pipeline.go
:WhyStopped()
methodjoinError()
instead ofsetError()
to the pipeline's error propertyStart()
and introducing methodsImportHandler()
,ProcessorHandler()
, andExporterHandler()
conduit/pipeline/pipeline_bench_test.go
- benchmarker for the pipeline that includes sleeping plugins with an importer, 2 processors, and an exporter.pkg/cli/cli.go
: remove a line break from the final error printoutIssues
118
TODO
PipelineData
inside ofBlockData
OStart()
and any resulting orphans inpipeline.go
master
which includes 30 second timeout #133WhyStopped()
?~ NOT FOR NOW. Reconsider this for #97internal-tools/.../logstats.go
be modified?~ This is under the umbrella of the "Logging Plugin Performance" threadTesting
E2E
pipeline_bench_test.go
Running a new benchmark test twice on the original code and the new, we have the following results. Note the most pertinent results for the typical indexer DB population use case is
exporter_10ms_while_others_1ms
:Block Generator Results
Running the block generator test using
SCENARIO = scenarios/config.allmixed.small.yml
for 30s, with the original code and the new, each time for 2 experiments we have:Local test network 5 minute sprint
I used the Justfile command
to bootstrap testnet and run a postgresql exporter against it for 300 seconds. I ran it a number of times against both the original pipeline and the new one. Here are the experimental results:
On EC2 - CLASSIC vs. PIPELINING vs. 30 Second Timeout vs. FINAL
I ran catchup tests for 4 versions of conduit:
master
at the time of the runpipelining
branch at the same timepipelining
branch after a 30 second timeout was introduced in the algod importerThere are much more detailed results in a google sheets document, but the summary is:
SUMMARY