[Support Request]: Incorrect/Missing Goals and Progress Data in Active Plan

pongthepm commented 1 year ago

Hi all, The BVBD user has reported a problem as below. Could you please fix it? Thank you, Pongthep

Plan: A2 ปลายลำไต (9006060108) 2023-05-26_[จับยุง+ตักลูกน้ำ] ID: 2ccf80c3-a622-55c9-9c6e-21f01569410b User: vbdu_12.2.3-1 (จ.สงขลา)

Problem details: The user successfully created the plan and set the goals (Image 1).

Then saved + sync in the mobile app. 1) HH registered 11/10 2) Scooped water larvae 3/2 3) catched mosquitoes 6/6

The data shown in the web is not correct (Image-2). 1) HH registration = 11/2 (550%) (incorrect: the target goal reduced from 10 to 2) 2) Scooping water larvae 3/2 (correct) 3) mosquitoe catching 3/6 (incorrect: 6 saved, but 3 shown) 4) no data showing on the map when clicking activities.

Data on Metabase dashboard is correct (image-3) 1) HH registered = 11 2) Scooped water larvae = 3 3) mosquitoes catching = 6

@truongnt @AngelaKabari

AngelaKabari commented 1 year ago

Hey @pongthepm I confirm receipt of this ticket.

@truongnt - Kindly provide the go ahead for Ona to action this?

truongnt commented 1 year ago

@AngelaKabari Pls help to fix the issue

AngelaKabari commented 1 year ago

Thank you for the authorisation Truong.

@HusnaHariz please take a look at this issue. Find below some additional information: Plan name: A2 tip of the trunk (9006060108) 2023-05-26_[catching mosquitoes + scooping larvae] investigation Plan ID: 2ccf80c3-a622-55c9-9c6e-21f01569410b User: vbdu_12.2.3-1 Map and progress view: https://mhealth.ddc.moph.go.th/focus-investigation/map/9c0e32c8-4b8b-5da3-889b-0de8e3aeefe6 Plan view page: https://mhealth.ddc.moph.go.th/focus-investigation/view/0e3272b2-2e7f-4756-85c4-a9be6313f2b9 Action plan goals view page: https://mhealth.ddc.moph.go.th/plans/update/2ccf80c3-a622-55c9-9c6e-21f01569410b

HusnaHariz commented 1 year ago

The progress view uses data from the targets_materialized_view, which is as follow:

Data from the database,

reveal1=> select plan_id, goal_id, goal_value, goal_unit, task_count, completed_task_count, task_business_status_map from  thailand_prod.targets_materialized_view where plan_id = '2ccf80c3-a622-55c9-9c6e-21f01569410b';
               plan_id                |        goal_id         | goal_value |   goal_unit    | task_count | completed_task_count |        task_business_status_map
--------------------------------------+------------------------+------------+----------------+------------+----------------------+----------------------------------------
 2ccf80c3-a622-55c9-9c6e-21f01569410b | Larval_Dipping         |       2.00 | activit(y|ies) |          3 |                    3 | { "Complete" : 3 }
 2ccf80c3-a622-55c9-9c6e-21f01569410b | RACD_register_families |      10.00 | Percent        |         15 |                   11 | { "Complete" : 11, "Not Visited" : 4 }
 2ccf80c3-a622-55c9-9c6e-21f01569410b | Mosquito_Collection    |       6.00 | activit(y|ies) |          6 |                    3 | { "Complete" : 3, "Not Visited" : 3 }
(3 rows)

The goal_value for each goal is similar to what has been entered in the webapp.

Followed up with @p-netm to determine how the progress view determines the level of progress and the following has been noted.

For HouseHolds, the goal targets entered , 10, represents a percentage of the total households (task_count). The progress is thus calculated as (15 * 10/100), which is is then rounded up to 2 as indicated in the progress bar. Here is a link to the reveal web code where this calculation is done.
We have also noted that the map now displays data indicated by the pink dots. No update was done on our end. It's not clear why the map was blank before. The missing data may have been due to a sync delay. Is this the data that was expected on the map?

For mosquito caching, as per this slice (data shown below), the completed_task_count for Mosquito_Collection is 3, which is what the progress map uses to display.

records": [
      {
        "id": "c1eedff7-a551-557f-a5bf-5ceae7c7d6ab",
        "plan_id": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "jurisdiction_id": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "goal_id": "RACD_register_families",
        "measure": "ร้อยละของบ้าน/สิ่งปลูกสร้างที่ได้ลงทะเบียนข้อมูลครัวเรือน",
        "goal_value": 10.0,
        "goal_comparator": ">=",
        "goal_unit": "Percent",
        "action_code": "RACD Register Family",
        "action_description": "ลงทะเบียนครัวเรือนและสมาชิกในครัวเรือนภายในพื้นที่ปฏิบัติงาน",
        "action_prefix": "1",
        "action_reason": "Routine",
        "action_title": "ลงทะเบียนครัวเรือน",
        "task_count": 15,
        "completed_task_count": 11,
        "task_business_status_map": "{\"Complete\": 11, \"Not Visited\": 4}"
      },
      {
        "id": "86630347-0539-5d9e-a265-8a4ac2cdd757",
        "plan_id": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "jurisdiction_id": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "goal_id": "Larval_Dipping",
        "measure": "จำนวนกิจกรรมการตักลูกน้ำ",
        "goal_value": 2.0,
        "goal_comparator": ">=",
        "goal_unit": "activit(y|ies)",
        "action_code": "Larval Dipping",
        "action_description": "ดำเนินกิจกรรมจับลูกน้ำในพื้นที่ปฏิบัติงาน",
        "action_prefix": "2",
        "action_reason": "Investigation",
        "action_title": "กิจกรรมการตักลูกน้ำ",
        "task_count": 3,
        "completed_task_count": 3,
        "task_business_status_map": "{\"Complete\": 3}"
      },
      {
        "id": "fb83781d-0502-5567-9028-fdda06e0fba7",
        "plan_id": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "jurisdiction_id": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "goal_id": "Mosquito_Collection",
        "measure": "จำนวนกิจกรรมการจับยุง",
        "goal_value": 6.0,
        "goal_comparator": ">=",
        "goal_unit": "activit(y|ies)",
        "action_code": "Mosquito Collection",
        "action_description": "ดำเนินกิจกรรมจับยุงในพื้นที่ปฏิบัติงาน",
        "action_prefix": "3",
        "action_reason": "Investigation",
        "action_title": "กิจกรรมการจับยุง",
        "task_count": 6,
        "completed_task_count": 3,
        "task_business_status_map": "{\"Complete\": 3, \"Not Visited\": 3}"
      }
    ]

As per the data in the canopy db, a total of 6 tasks exist but 3 are yet to be completed. Will be looking to understand why these are still marked as Not Visited.

HusnaHariz commented 1 year ago

The data is marked Not Visited, because this is the raw data in the raw_tasks and task tables as fetched from opensrp. Therefore, all matviews are refreshing against this data. Unclear if this is to be updated at somepoint

reveal1=> select full_json ->> 'code', full_json ->> 'businessStatus',full_json ->> 'status', date_created, last_updated from thailand_prod.raw_tasks where full_json::json ->> 'planIdentifier' = '2ccf80c3-a622-55c9-9c6e-21f01569410b' and full_json::json ->> 'code' = 'Mosquito Collection';
      ?column?       |  ?column?   | ?column?  |    date_created     |        last_updated
---------------------+-------------+-----------+---------------------+----------------------------
 Mosquito Collection | Not Visited | Ready     | 2023-05-29 15:55:00 | 2023-05-29 15:55:02.193963
 Mosquito Collection | Not Visited | Ready     | 2023-05-29 15:55:00 | 2023-05-29 15:55:02.193992
 Mosquito Collection | Not Visited | Ready     | 2023-05-29 15:55:00 | 2023-05-29 16:10:40.17625
 Mosquito Collection | Complete    | Completed | 2023-05-30 10:45:27 | 2023-05-30 16:05:34
 Mosquito Collection | Complete    | Completed | 2023-05-30 10:55:27 | 2023-05-30 16:05:34
 Mosquito Collection | Complete    | Completed | 2023-05-30 10:45:27 | 2023-05-30 16:05:34
(6 rows)

reveal1=> select created_at, plan_identifier, status, business_status, priority, code, description from thailand_prod.tasks where plan_identifier = '2ccf80c3-a622-55c9-9c6e-21f01569410b' and code = 'Mosquito Collection';
         created_at         |           plan_identifier            |  status   | business_status | priority |        code         |            description
----------------------------+--------------------------------------+-----------+-----------------+----------+---------------------+------------------------------------
 2023-05-30 10:59:59.887805 | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Completed | Complete        | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
 2023-05-30 10:49:38.729475 | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Completed | Complete        | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
 2023-05-30 10:49:38.784074 | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Completed | Complete        | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
 2023-05-29 20:03:22.451076 | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Ready     | Not Visited     | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
 2023-05-29 20:03:22.496612 | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Ready     | Not Visited     | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
 2023-05-29 20:03:22.48326  | 2ccf80c3-a622-55c9-9c6e-21f01569410b | Ready     | Not Visited     | 3        | Mosquito Collection | กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล
(6 rows)

Data from opensrp https://servermhealth.ddc.moph.go.th/opensrp/rest/task/getAll?serverVersion=1601993604760&limit=1000

[
    {
        "executionStartDate": "2023-05-29T1549",
        "identifier": "d2b8d7ac-99ca-4097-bc5b-239a986d63ae",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Ready",
        "businessStatus": "Not Visited",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "e9c96a63-d9a9-46df-a745-2257d0cdba61",
        "authoredOn": "2023-05-29T2254",
        "lastModified": "2023-05-29T2254",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993604762,
        "syncStatus": "Created",
        "structureId": "e9c96a63-d9a9-46df-a745-2257d0cdba61"
    },
    {
        "executionStartDate": "2023-05-29T1552",
        "identifier": "8fedd835-ef18-49d8-b9c5-bcdc07f63a26",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Ready",
        "businessStatus": "Not Visited",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "1230c889-741b-4ca3-a1e0-a4933a248cae",
        "authoredOn": "2023-05-29T2254",
        "lastModified": "2023-05-29T2254",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993604764,
        "syncStatus": "Created",
        "structureId": "1230c889-741b-4ca3-a1e0-a4933a248cae"
    },
    {
        "executionStartDate": "2023-05-29T1549",
        "identifier": "d48a5eac-c969-448b-a30b-d08b77dbed97",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Ready",
        "businessStatus": "Not Visited",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "28fd8fed-ee53-4524-b7b2-23968129be20",
        "authoredOn": "2023-05-29T2254",
        "lastModified": "2023-05-29T2254",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993604765,
        "syncStatus": "Created",
        "structureId": "28fd8fed-ee53-4524-b7b2-23968129be20"
    },
    {
        "executionStartDate": "2023-05-30T1742",
        "identifier": "609aabd2-7439-4048-bafd-51c5b3ac075d",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Completed",
        "businessStatus": "Complete",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "151d4a68-efae-4cfe-b498-c52076c4ee64",
        "authoredOn": "2023-05-30T1743",
        "lastModified": "2023-05-30T2304",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993605309,
        "syncStatus": "Created",
        "structureId": "151d4a68-efae-4cfe-b498-c52076c4ee64"
    },
    {
        "executionStartDate": "2023-05-30T1742",
        "identifier": "09708a43-f69a-4771-a9f4-3e5c417202c2",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Completed",
        "businessStatus": "Complete",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "38122fdc-05c6-4bdf-9d96-dcad1256f6ac",
        "authoredOn": "2023-05-30T1743",
        "lastModified": "2023-05-30T2304",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993605310,
        "syncStatus": "Created",
        "structureId": "38122fdc-05c6-4bdf-9d96-dcad1256f6ac"
    },
    {
        "executionStartDate": "2023-05-30T1744",
        "identifier": "3c659062-af10-43a0-873e-1480d57ed2c5",
        "planIdentifier": "2ccf80c3-a622-55c9-9c6e-21f01569410b",
        "groupIdentifier": "0e3272b2-2e7f-4756-85c4-a9be6313f2b9",
        "status": "Completed",
        "businessStatus": "Complete",
        "priority": 3,
        "code": "Mosquito Collection",
        "description": "กำหนดสถานที่เก็บตัวอย่างยุง และการบันทึกผล",
        "focus": "dd44889f-a870-5b0a-920a-be90219f21cb",
        "for": "84478660-9e82-427b-9d1a-9206a5014d77",
        "authoredOn": "2023-05-30T1752",
        "lastModified": "2023-05-30T2304",
        "owner": "vbdu_12.2.3-1",
        "serverVersion": 1601993605311,
        "syncStatus": "Created",
        "structureId": "84478660-9e82-427b-9d1a-9206a5014d77"
    }
]

AngelaKabari commented 1 year ago

Hello @pongthepm - Please ask DVBD to look at the map now and they will see that all 6 mosquito catching activities are showing up on the map. The progress is showing as 3 out of 6 (50%) because 3 of the mosquito catching points have the status "done" (green dots) while the other 3 mosquito catching points have the status “not visited” (yellow dots). If the team on the ground visit the sites then update the status on the 3 mosquito catching points then they will move from “not visited” to “done”, and the progress bar will show 6 out of 6 (100%).

AngelaKabari commented 1 year ago

@HusnaHariz and @p-netm Please note that the original goal for Household registration was 10, which is what is showing here. But for some reason Reveal Web progress page is displaying the goal target as 2. So what should display on Reveal Web should be 11/10 (110%) and not 11/2 (550%).

Can you please look into why this is happening?

AngelaKabari commented 1 year ago

Both User vbdu_12.2.3-1 and I misunderstood how the household registration progress is calculated. Find below an excellent explanation from @p-netm:

Goal targets can either be an absolute numeric value or a relative percentage of total tasks generated, Have a look at the initial discussions on the web FI planning feature here. In column J of this data dictionary,one can see what form the different activity goal targets take. When creating a plan on the frontend, one can see what goal unit will be used for the activity. The attached screenshot shows that in the original EN text the goal unit for house hold registration is a percent. I believe it should read the same when in Thai, but if not, then the translation strings on web would need to be updated. Therefore to iterate what Husna said, the data we have for house hold registration tasks is both stored and interpreted correctly by the Reveal system.

@pongthepm please confirm that this issue has been satisfactorily resolved and so we can close it out? Thanks!

pongthepm commented 1 year ago

Thank you for your support @AngelaKabari. I have notified BVBD to check it. I will update you the result.

pongthepm commented 1 year ago

Hi,

The BVBD user informed me that she can see 6 green points off the map of mosquitoes catching correctly. However, she said the values of HH registration and mosquito catching are still incorrect. The progress value of mosquito catching should be 6/6. In terms of HH registration target, do you mean the user entered the goal unit in percentage somehow? From the screen captured she sent me, the progress value should be 11/10 not 10/2, right?

Thank you, Pongthep

AngelaKabari commented 1 year ago

Hello @pongthepm,

Please see this comment for the explanation of the mosquito catching progress
Yes, the user entered the goal unit as a relative percentage of total tasks generated not an absolute numeric value as you can see here:

Please revisit the explanations given within this comment as well as this comment to understand why the progress value is 11 of 2 houses instead of 11 of 10 houses.

AngelaKabari commented 1 year ago

Hello @pongthepm, please confirm that this issue has been satisfactorily resolved so that we can close it out?

pongthepm commented 1 year ago

Hi Angela, You can close this issue. Thank you, Pongthep

onaio / Reveal-Thailand

[Support Request]: Incorrect/Missing Goals and Progress Data in Active Plan #33