search

m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

storage-rw related throughput behavior? Dashboard improvement? #984

Closed gfr10598 closed 3 years ago

gfr10598 commented 3 years ago

When etl lacked storage-rw scope, the pipeline processed files much faster, but is unclear why. The parser has to do all of the same work, EXCEPT for the final json file write to the etl-mlab-xxx bucket. While this write might take some wall time, it should take negligible CPU time. These files are uncompressed JSONL files.

UNLESS the write failure causes the transcription to JSON to terminate, or stops the parsing after the first attempt to write the row buffer.

In any case, these problems should be better communicated to gardener, and show up clearly in the dashboard.

See https://grafana.mlab-sandbox.measurementlab.net/goto/IUGcvWuGz

Screen Shot 2021-04-11 at 9 15 48 AM
gfr10598 commented 3 years ago

Write failures are now included in the new gfr/pipeline-overview dashboard.

The faster processing is due to early termination of the tasks when there are widespread write errors, not an optimization opportunity.