WCARO DRC: Dashboard Data Not Updated

[ageryck]

Background: The client reported dashboard data do not reflect data collected as recent as Oct 2022, we need to investigate the data in the OpenSRP server and make comparative queries in canopy DB to ascertain we have no pipeline issues, in case we have a pipeline breakdown we will seek canopy support to resolve the same. Some action items are outlined below;

• [x] Query OpenSRP DB (opensrp_wcaro_drc) to ascertain we have events & clients collected and synced for as recent as the month of Oct 2022
• [x] Query Canopy DB(wcaro_project) for the same records in the query above
• [x] Sync problems reported in Mikonga region, query latest records synced to OpenSRP DB from teams in Mikonga region. We will share data sync details with the client (latest data synced in the region by which providers)
• [x] Documents query findings All access details in BW WCARO DRC Collection

[hilpitome]

A query of the OpenSRP database shows that we have events for as recently as 21st October 2022

[hilpitome]

However, the number of events from the beginning of June till date seem relatively low at 513 (from 2022-06-01 to today) compared to 9773 from January. It can also be noted that there were 360 events created on 21st October 2022.

[hilpitome]

From the Mikonga region, we had 16 events created on 2022-09-14 from the user mikonga11

[joyce-x-chen]

Thanks, @hilpitome.

However, the number of events from the beginning of June till date seem relatively low at 513 (from 2022-06-01 to today) compared to 9773 from January. It can also be noted that there were 360 events created on 21st October 2022.

Is that the date the 360 events were synced or the date that those records were created on the device?

[ageryck]

@hilpitome help query canopy DB to eliminate suspicion of pipeline breakdown then we can get back to the client these findings

[hilpitome]

Thanks, @hilpitome.

However, the number of events from the beginning of June till date seem relatively low at 513 (from 2022-06-01 to today) compared to 9773 from January. It can also be noted that there were 360 events created on 21st October 2022.

Is that the date the 360 events were synced or the date that those records were created on the device?

@joyce-x-chen That is the date that those events were synced to the server

[hilpitome]

@ageryck is there an ongoing issue on canopy on this issue, or do you have someone in the canopy team that we can work with? I am not familiar with the canopy db cc @joyce-x-chen

[ageryck]

@hilpitome no reported issue, the canopy DB is another Postgres where the pipeline sink the data from OpenSRP and dashboard queries are run on it. If we confirm the number of the most recent data look-alike it eliminates possible pipeline processing issues and we can be confident the dashboard shows the most recent data. The creds are in BW but feel free to reach out to @lincmba if stuck

[lincmba]

@ageryck @hilpitome

The data pipelines are running fine with no errors. The data seems to be streaming in fine, I can get clients created as recently as 21st as below

wcaro_project=> select date_created from clients order by date_created limit 5;
      date_created
-------------------------
 2020-03-01 01:03:15.853
 2020-03-01 01:03:33.266
 2020-03-01 01:05:06.848
 2020-03-01 01:06:01.601
 2020-03-01 01:06:19.036
(5 rows)

However, I noticed we only have clients, locations and jurisdictions connectors. The events connector has not been built.

[ageryck]

@ageryck @hilpitome

The data pipelines are running fine with no errors. The data seems to be streaming in fine, I can get clients created as recently as 21st as below

wcaro_project=> select date_created from clients order by date_created limit 5;
      date_created
-------------------------
 2020-03-01 01:03:15.853
 2020-03-01 01:03:33.266
 2020-03-01 01:05:06.848
 2020-03-01 01:06:01.601
 2020-03-01 01:06:19.036
(5 rows)

However, I noticed we only have clients, locations and jurisdictions connectors. The events connector has not been built.

@lincmba is this DRC specific data? Are the pipelines running on beam or nifi ? we should have events pipeline deployed

[lincmba]

@ageryck

Upon further investigations, I found the events ETL. The jurisdictions, locations and clients pipelines were created on beam. The events connector was created on nifi. The nifi events connector is running fine with no errors. I can see in events in the raw_events table created as recent as 21st October 2022. This is consistent with the results @hilpitome provided (on the most recent event/client). The following is a sample sql query

wcaro_project=> select full_json->'dateCreated' from raw_events order by full_json->'dateCreated' desc limit 5;
          ?column?
----------------------------
 "2022-10-21T08:34:07.359Z"
 "2022-10-21T08:34:07.351Z"
 "2022-10-21T08:34:07.342Z"
 "2022-10-21T08:34:07.308Z"
 "2022-10-21T08:34:07.274Z"
(5 rows)

wcaro_project=>

[ageryck]

closed untill client shared additional information

[ageryck]

No additional info shared todate