search

onaio / Reveal-Thailand

Repository to raise & track issues, share code and documentation, and manage the Reveal Thailand Handover and Support Project (2022-2023)
0 stars 0 forks source link

[Support Request] Data from plan is not syncing to web and dashboard #1

Closed AngelaKabari closed 1 year ago

AngelaKabari commented 1 year ago

What is the nature of this Technical Support Request? Please describe the need/requirement in detail

(A clear and concise description of what the problem is)

A member of the DVBD field staff is experiencing a problem with a plan id: 5bc3f9c7-5763-5645-a6df-f457c86f03dd with user: vbdu_2.1.1-1. They filled in the data, then sync completed, but there is no data on the web and dashboard.

Describe the solution you'd like

(A clear and concise description of what you want to happen)

Describe alternatives you've considered

None.

Additional context

(Add any other context or screenshots about the request here)

None.

AngelaKabari commented 1 year ago

The data was synced manually using Postman but the dashboard does not show any data. Below are the links for the plan reports:

We will need to get help from someone on the Canopy team to understand if the data was synced to canopy and why it does not show up on the web. I can confirm that the data is present in the OpenSRP database

cc @AngelaKabari

Originally posted by @ekigamba in https://github.com/opensrp/support/issues/272#issuecomment-1347906516

HusnaHariz commented 1 year ago

Plan has been synced from opensrp to the canopy tables, both the raw and transactional tables

reveal1=> select count(*) from thailand_prod.raw_plans where id = '5bc3f9c7-5763-5645-a6df-f457c86f03dd';
 count
-------
     1
(1 row)
reveal1=> select count(*) from thailand_prod.plans where identifier = '5bc3f9c7-5763-5645-a6df-f457c86f03dd';
 count
-------
     1
(1 row)
reveal1=> select count(*) from thailand_prod.plans_materialzied_view where plan_id = '5bc3f9c7-5763-5645-a6df-f457c86f03dd';
 count
-------
     1
(1 row)

Checking if the slice used in the webapp includes the plan

HusnaHariz commented 1 year ago

@ekigamba Is there a filter in the tasks endpoint that allows one to retrieve tasks related to a specific plan in this case 5bc3f9c7-5763-5645-a6df-f457c86f03dd

ekigamba commented 1 year ago
HusnaHariz commented 1 year ago

Tasks are in sync between the opensSRP and Canopy DBs

OpenSRP DB

opensrp=# select count(*) from core.task where json ->> 'planIdentifier' = '5bc3f9c7-5763-5645-a6df-f457c86f03dd';
 count
-------
     1
(1 row)

Canopy DB

reveal1=> select count(*) from thailand_prod.tasks where plan_identifier ='5bc3f9c7-5763-5645-a6df-f457c86f03dd' ;
 count
-------
     1
(1 row)
ekigamba commented 1 year ago

Thanks @HusnaHariz I will be manually uploading the tasks from the user given the new information

ekigamba commented 1 year ago 
    The client database was provided and only 531 of 534 events are marked as synced

There are other issues noted:

Originally posted by @ekigamba in https://github.com/opensrp/support/issues/272#issuecomment-1333720141

ekigamba commented 1 year ago

I have uploaded 190 tasks to the OpenSRP server using Postman Endpoint {{url}}/opensrp/rest/v2/task/add POST

tasks_users.txt

Queries used to extract data from the DB:

Canopy needs to pull the tasks and build the materialized view from which the API makes the query calls. Canopy pulls data from OpenSRP every 5 minutes but the materialized view takes time to build and might require to be refreshed according to @HusnaHariz

AngelaKabari commented 1 year ago

Thanks for this @ekigamba. I'll check tomorrow and see whether the tasks are now visible on the dashboard.

AngelaKabari commented 1 year ago

Hello @ekigamba I confirm that I can now see I confirm that I can see the progress on map view page (https://mhealth.ddc.moph.go.th/focus-investigation/map/7015b227-1a4a-56b9-b5c4-9c60e4b151db). However, when I look at the plan view page (https://mhealth.ddc.moph.go.th/focus-investigation/view/5bc3f9c7-5763-5645-a6df-f457c86f03dd) I get the error ”An error occurred, please try refreshing the page. The specific error is : Could not load points on map.” (see below screenshot). The view page also isn’t loading the plans and activities.

Could not load points error

@HusnaHariz please look into the above error and let me know what’s causing it, and what the fix would be?

AngelaKabari commented 1 year ago

I have also noted that while the map view page is displaying the progress towards goals, the GPS points aren't loading and showing on the map.

See below an example screenshot of what should be displayed:

points showing

vs. what is being displayed on the map view page:

No points showing
ekigamba commented 1 year ago

Data that is synced on Reveal Thailand

cc @AngelaKabari

This means that I need to upload the structures from the users data dump

AngelaKabari commented 1 year ago

Thanks for the update @ekigamba. Let me know when this is complete?

ekigamba commented 1 year ago

@AngelaKabari I have uploaded 99 structures to Reveal for this user. This should have reflected by tomorrow morning

AngelaKabari commented 1 year ago

Hello @ekigamba I've just checked the Reveal Web Dashboard and the structures are still not displaying in both focus investigation plan pages (views and map). Kindly take a look at it again?

ekigamba commented 1 year ago

All the structures are in the Reveal production database

AngelaKabari commented 1 year ago

Thanks @ekigamba let me loop in the OpenSRP Web Team to help troubleshoot this.

peterMuriuki commented 1 year ago
  1. The single Active FI map view on page: https://mhealth.ddc.moph.go.th/focus-investigation/map/7015b227-1a4a-56b9-b5c4-9c60e4b151db does not load/display points on the map. This data is computed from the tasks slice 8.

The request to fetch the tasks data fails with a 403 error.

https://supersetmhealth.ddc.moph.go.th/superset/slice_json/8?form_data={%22adhoc_filters%22:[{%22clause%22:%22WHERE%22,%22expressionType%22:%22SQL%22,%22sqlExpression%22:%22(plan_id%20=%20%275bc3f9c7-5763-5645-a6df-f457c86f03dd%27)%20OR%20\n%20%20%20%20%20%20%20%20%20%20%20%20(jurisdiction_id%20=%20%2794c94f23-568a-4467-8959-5d02b8f9f49f%27%20AND%20goal_id%20=%20%27Case_Confirmation%27)%22}],%22row_limit%22:250}

Running it on the browser yields this view: Screenshot from 2023-01-17 09-26-52

@AngelaKabari I understand we do not manage the firewall, I would suggest reaching to the client's technical team to confirm why they firewall fails this request. cc @bennsimon .

  1. The plan view page, or rather the FI jurisdiction view, displays a list of FI plans for a given operational area. I am not sure what steps were taken to arrive at this url: https://mhealth.ddc.moph.go.th/focus-investigation/view/5bc3f9c7-5763-5645-a6df-f457c86f03dd, but I think its malformed, the id at the end of the url should be that of a jurisdiction and not a plan. I think the correct url, which is what the web app would direct you to is https://mhealth.ddc.moph.go.th/focus-investigation/view/94c94f23-568a-4467-8959-5d02b8f9f49f.
AngelaKabari commented 1 year ago

@p-netm I shall check with DVBD what's going on with the firewall as it is also created #15 where routine action plans are not being generated.

Once that is resolved then we can pick this up.

AngelaKabari commented 1 year ago

Hello @p-netm the Reveal Thailand system is back online so let us pick this us again next week.

peterMuriuki commented 1 year ago

@AngelaKabari

cc @HusnaHariz

AngelaKabari commented 1 year ago

Thanks for getting to the root of this. @p-netm I think the missing geometry in the geojsons for the tasks might be stemming from the manual addition of those tasks to the database from the user's data dump. Let me consult @Rkareko and @ekigamba on this before we loop @HusnaHariz back in.

ekigamba commented 1 year ago

@AngelaKabari I can confirm that I uploaded the structures containing the geojson. The tasks reference these structures.

Is there someone who can explain why this was happening?

On the map page mhealth.ddc.moph.go.th/focus-investigation/map/7015b227-1a4a-56b9-b5c4-9c60e4b151db, I can see that the page is loading the 99 structures from the network requests but they are still not displaying once you click on the household registration

cc @Rkareko

Wambere commented 1 year ago

@ekigamba is it possible to share some of the manually uploaded tasks ans structure ids, so we can try track them in the warehouse?

peterMuriuki commented 1 year ago

Had a chance to look at this yesterday with a lot of help from @HusnaHariz @lincmba and @ekigamba . Let me know if I missed anything or need to add more contextual info.

so what we knew:

image

cc @AngelaKabari

AngelaKabari commented 1 year ago

Thanks @p-netm both for your hard work and for this very comprehensive explanation. I appreciate the level of detail. Thank you too @ekigamba, @lincmba, and @HusnaHariz for your valuable input into troubleshooting and resolving this.

I have checked and the structures are displaying as they should on Reveal Web.

@pongthepm kindly ask DVBD to confirm that this has been satisfactorily resolved so that we can close it out?