m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

tcpinfo: consider minimizing snapshot thinning #1068

Open stephen-soltesz opened 2 years ago

stephen-soltesz commented 2 years ago

Originally, tcpinfo snapshots were thinned 10:1, leaving what could be up to 10ms resolution to 100ms for NDT measurements. For more detailed analysis this higher resolution may be preferable. However, the rows will be larger, BQ storage more expensive and queries more costly. So, we want to strike a balance by revising the simple 10:1 implementation with one that meets needs for analysis.

See also:

stephen-soltesz commented 1 year ago

Also related to: