Production data types have 'max-min' parse times older than historical cycle time

[stephen-soltesz]

In production monitoring, even though gardener processes dates in sync, the datatypes have different max-min parser times.

This is unexpected. The cause is unknown.

[stephen-soltesz]

SELECT  MIN(parser.Time) mintime, parser.ArchiveURL
from `mlab-oti.ndt_raw.ndt7`
WHERE parser.Time between '2022-04-10' and '2022-05-01'
GROUP by parser.ArchiveURL
order by mintime desc

Row | mintime | ArchiveURL |  
1   | 2022-04-12 00:39:48.663516 UTC | gs://archive-measurement-lab/ndt/ndt7/2021/10/16/20211016T072521.128809Z-ndt7-mlab2-mia05-ndt.tgz |  
2   | 2022-04-11 23:42:51.021691 UTC | gs://archive-measurement-lab/ndt/ndt7/2021/10/14/20211014T050728.240194Z-ndt7-mlab1-sin01-ndt.tgz |  
3   | 2022-04-11 23:17:19.303483 UTC | gs://archive-measurement-lab/ndt/ndt7/2021/10/13/20211013T152757.265738Z-ndt7-mlab1-hnd05-ndt.tgz |  
4   | 2022-04-11 23:14:53.150130 UTC | gs://archive-measurement-lab/ndt/ndt7/2021/10/13/20211013T131052.627372Z-ndt7-mlab3-mnl02-ndt.tgz

$ gsutil cp gs://etl-mlab-oti/ndt/ndt7/2021/10/16/20211016T072521.128809Z-ndt7-mlab2-mia05-ndt.tgz.json .
$ jq '.Parser.Time'  20211016T072521.128809Z-ndt7-mlab2-mia05-ndt.tgz.json  | sort -r

There are no Parser.Time values that match the parser.Time in BQ.

[stephen-soltesz]

Why wouldn't these rows be reset after a full cycle?

Speculation:

• perhaps rows/partitions that should be deleted/overwritten are being skipped/missed by gardener, and linger in BQ
• perhaps this may occur during a deployment, when gardener is killed during a job
• perhaps this may occur during a load operation, when gardener is killed during a job