Tuning Sidestream & Traceroute parsers

m-lab / etl

M-Lab ingestion pipeline

Apache License 2.0

22 stars 7 forks source link

Tuning Sidestream & Traceroute parsers #459

Open stephen-soltesz opened 6 years ago

stephen-soltesz commented 6 years ago

The new combined dashboard revealed some insights that we may have missed before:

typical duration for pt & ss handlers is ~5-20mins
this may cause some additional storage related errors for both services
the effective processing rates for pt & ss appear to be about 0.5/sec
more tuning of the BQBufferSize may help
traceroute parser reports higher than expected ParseAndInsertErrors

@yachang fyi

yachang commented 6 years ago

pt & ss worker handle tar file which contains thousands of tests each, so it is expected to take longer.
the average BQ insertion time used for PT is about 350ms, for ss, about 1ms, for NDT, about 200ms.
BQBufferSize is NDT: 10, SS: 100, PT: 300, I will try to reduce the Buffer size for PT.

stephen-soltesz commented 6 years ago

Increasing the buffer size will mean that additional rows are written at once. A larger buffer may help increase the BQ insertion time (because more rows are inserted per call) while decreasing the overall processing time.