search

opensrp / opensrp-reveal-datawarehouse

Scripts that build the Reveal data warehouse
0 stars 1 forks source link

Check Beam and NiFi sync setup for Namibia, add additional Namibia monitoring for count totals #55

Open pld opened 3 years ago

pld commented 3 years ago

Beam Dashboard is here https://graylog.onalabs.org/dashboards/5f3e87e8fb60215f281b6947

From @gstuder-ona:

note that the numbers on that dashboard get reset after every restart so they aren't indicative of totals

gstuder-ona commented 3 years ago 
reveal1=> select count(*) from namibia_prod.tasks;
 count 
-------
 46556
(1 row)
opensrp=> select count(*) from core.task;
 count 
-------
 46556
(1 row)

https://graylog.onalabs.org/dashboards/5f3e87e8fb60215f281b6947

sync_status

Note 1) that the sync doesn't actually seem to have ever been restarted, which means the counts are since 0 and so comparable to the db counts and 2) the sync stats count all upserts, and so may have written more tasks if some were updated. It's hard to tell without deeper auditing - maybe we could add a Beam option to count inserts/upserts independently?

gstuder-ona commented 3 years ago

@pld @craigappl lmk if more flow/db auditing is necessary but at this point it seems like the beam ETL has been working correctly for 30 days straight with no errors on 200MB of RAM