Flowminder / FlowKit

FlowKit: Flowminder CDR analytics toolkit
https://flowminder.github.io/FlowKit/
Mozilla Public License 2.0
86 stars 20 forks source link

IMSI/IMEI QA checks should ignore nulls #6619

Closed jc-harrison closed 2 months ago

jc-harrison commented 2 months ago

The FlowETL QA checks count_imsis, count_imeis, max_msisdns_per_imsi and max_msisdns_per_imei count NULL as a valid IMEI/IMSI value. This is particularly problematic for max_msisdns_per_imsi/max_msisdns_per_imei, since these checks will return a very large count if a portion of the events are missing an imei or imsi.

These checks should instead operate only on events with non-null imei/imsi. We have other checks to count the rows where these fields are null.