Add utilities for performing routine consistency checks of duplicate entries in obs_* tables

ooni / data

OONI Data CLI and Pipeline v5

https://docs.ooni.org/data

8 stars 4 forks source link

Add utilities for performing routine consistency checks of duplicate entries in obs_* tables #91

Closed hellais closed 2 months ago

hellais commented 2 months ago

If we rely on de-duplication for handling obs_* table row deduplication, since the observation count might decrease in new runs, we should make sure that some duplicate rows are not left around.

See: https://github.com/ooni/data/issues/88 & https://github.com/ooni/data/issues/87

hellais commented 2 months ago

This was done as part of https://github.com/ooni/data/pull/97