m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

Prevent scamper1 rows that are too large #1133

Closed stephen-soltesz closed 1 month ago

stephen-soltesz commented 1 month ago

Periodically, daily jobs fail due to errors like those below. Sometimes scamper1 generates rows that are greater than 100MB when the maximum supported Load row size is 100MB, leading to these failures. As well, since the parser/gardner system does not retry failed daily jobs, one large file prevents the whole day from loading successfully.

This change simply filters raw traces larger than 90MB to prevent this type of failure. It is an open question why scamper would ever generate a trace this large. This is likely to be an anomalous measurement already. This change does not address the root cause.

20241018:ndt/scamper1 Load {Location: "gs://etl-mlab-oti-us-central1/archive-measurement-lab/ndt/scamper1/2024/10/18/20241018T140116.143544Z-scamper1-mlab1-gru03-ndt.tgz.jsonl"; 
Message: "Error while reading data, error message: JSON parsing error in row starting at position  116704618:
Row size is larger than: 104857600. File: gs://etl-mlab-oti-us-central1/archive-measurement-lab/ndt/scamper1/2024/10/18/20241018T140116.143544Z-scamper1-mlab1-gru03-ndt.tgz.jsonl"; Reason: "invalid"}

See also: https://github.com/m-lab/dev-tracker/issues/744


This change is Reviewable

coveralls commented 1 month ago

Pull Request Test Coverage Report for Build 7551

Details


Changes Missing Coverage Covered Lines Changed/Added Lines %
parser/scamper1.go 2 5 40.0%
<!-- Total: 2 5 40.0% -->
Files with Coverage Reduction New Missed Lines %
active/active.go 4 88.54%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 7548: -0.1%
Covered Lines: 3338
Relevant Lines: 4963

💛 - Coveralls