search

m-lab / traceroute-caller

A sidecar service which runs traceroute after a connection closes
Apache License 2.0
18 stars 5 forks source link

daily tests data drop over past 10 days #68

Open yachang opened 4 years ago

yachang commented 4 years ago
2 2020-01-26 8539584  
3 2020-01-25 8802175  
4 2020-01-24 8804368  
5 2020-01-23 8989835  
6 2020-01-22 9547450  
7 2020-01-21 9553054  
8 2020-01-20 9684823  
9 2020-01-19 9787798  
10 2020-01-18 10059376  
11 2020-01-17 9847430  
12 2020-01-16 10236158  
13 2020-01-15 10198442  
14 2020-01-14 10056294  
15 2020-01-13 10208497
yachang commented 4 years ago

Not sure it is caused by delay from Gardener, or other unknown reason

yachang commented 4 years ago

iad drop more than 25% from 01/14 to 01/28 (about 2 weeks). Escalate this issue to P0

yachang commented 4 years ago

gsutil ls -l gs://archive-mlab-oti/ndt/traceroute/2020/01/14/iad totally 75M gsutil ls -l gs://archive-mlab-oti/ndt/traceroute/2020/01/27/iad totally 77M

The drop of tests in BQ table could be related to https://github.com/m-lab/etl-gardener/issues/204

yachang commented 4 years ago

Number of tests for 2020-02-11 increased back to 10017424 (about 10%)

Not sure it is a one day thing, will keep updating here.

stephen-soltesz commented 4 years ago

@yachang please add original query to help others reproduce.

yachang commented 4 years ago

https://github.com/m-lab/etl/issues/820

Is a issued triggered by this issue

yachang commented 4 years ago

here is the BQ used:

SELECT ts, COUNT(*) AS num

FROM ( SELECT DATE(TestTime) as ts FROM mlab-oti.base_tables.traceroute

WHERE DATE(_PARTITIONTIME) BETWEEN DATE("2020-01-01") AND DATE("2020-02-20") )

GROUP BY ts ORDER BY ts DESC