[HWF] Yes/Partial/No - Githubissues

:pushpin: References

Redmine: http://who-dev.essi.upc.edu:8083/issues/2065
Trello: https://trello.com/c/g5r5OsJq/20-yes-partial-no-issues

📓 Description

Yes/Partial/No check boxes are capturing data wrongly.

Back-end: No changes needed, we should not touch metadata because it's in PROD.
Front-end: We need to fix the HTML source to avoid the incorrect behavior.
Remapping: We need to create a script that cleans duplicated occurrences for a same dataValueSet and keep the last updated yes/partial/no option.

🔥 Issues found and solved

Exclusive checkboxes were only working in the front-end but DHIS2 has been storing non-exclusive options on all the modules.
When user changed period for the second time, the exclusive checkbox feature stopped working because the form had refreshed.
Two data elements on Module 3 were not treated as exclusive.
One data element on Module 8-10 was being treated as exclusive whilst it was a text entry field.
There are two additional inconsistencies in the metadata that should be reviewed in the future.

:bookmark_tabs: Tasks

[x] [DataEntryForm] Add new logic for exclusive yes/partial/no checkboxes (f9e9d70412b5c4217e73489bc3dc986a728d9c3c)
[x] [DataEntryForm] Fix bug where exclusive stopped working after changing period (6e6581617d772cdb2a3df69bef46791da61c2486)
[x] [Module 3] Fix two dataElements that were not being treated as exclusive (2603632a72434a6a6d91bcd25386d4dd0d94c0fc)
[x] [Module 8-10] Fix dataElement in that was incorrectly tagged as yes/partial/no (c06c85f681eabb941dcfa7510bcaf0c7144094e3)
[x] [Testing] Test that exclusive checkboxes apply correctly in /api/dataValueSets
[x] [Metadata] Investigate faulty dataElements and probable bugs
[x] [Script] Build a remapping script for duplicated data values

:memo: Others

Ocurrences found of yes/partial/no

#### Module 2 ``` var yespartialno = ["EtX7FWQ8HOf"]; ``` #### Module 3 ``` var yespartialno = [ "afgPWA9EO9d", "eaxvIzrZDyz", "RQZXBtwBToV", "h580rDibPjP", "c3tvWVyotti", "u06rnfoY988", "LTTnjIq6X6j", "HZci3ozLEL4", "QcMU3E5D2eN", "PPfHCBsbdGU", "gs8HWsMAFFl", "Km3h0ydFI2j", "fLT65oEdyRn", "GEDhlVgull2", "U5S2sbEnGT9", "o64uNjIuRXr", "XQgL1GHsnrT", "Dn2P7GXFCTr", "pnySnJ6om1A", "wTTVz32PAOn", "Qj97xn0ejxd", "nwRqq6w8MKq", "wqcSEURdLVe", "EGGz5kkdVNl", "l1U9oBgd2Pg", "JLq8DlnnFF9", "YDkIX98NArQ", "j4pHY4ToKPi", "Mx06wOF9Edi", "hJJOo4gRjQA", "GNErbB4poQO", "bq7cDa5kIgG", "Fu0P8tj6k3M", "v6O0oWTXY34", "VCZsq4EzHM1", "TYtubJjtDDs", "kxeieT5f8wy", "PG2MTqzwOAT", "VQk5aenfDR4", "pyMHfUIDXRf", "YVbwt0lb1lA", "m8xYEbXD3Ee", "pWAGDlaDi7y", "H4L8KbENVVk", "KPlsuTaezlc", "Pf77Sb5TYCR", "LJNl2upvs8S", "vKV2AVBhstz", "aEDrisPWh6i", "KcDPaDGLf9P", "zHNwRzmyOeL", "PE5XG7ZZgdc", "TdnUj1l3Z4i", "WsHpge8yVZM", "ubXH2Mko823", "KYNXNZlaQBc", "x9MXiBfrfFG", "wY9lL47ojb6", "OBs444qCCU9", "mHNrhvqWd3X", "cemAvj8TkMX" ]; ``` #### Module 4 ``` var yespartialno = ["Ybz3qUCWYKC"]; ``` #### Module 6 ``` var yespartialno = [ "lvs1RCQOOp4", "LYjT01KiTTw", "gGSJZCN2L8C", "FDUFnJmPb2t", "DVyhvgkJrXg", "Uyxr85SOJiI", "I5m9Dl43XO2", "gDinmRT6Pqb", "LUVRqQq2NLE" ]; ``` #### Module 8-10 ``` var yespartialno = [ "tKr3k3lbnJJ", "LW954y3FqL8", "ROVoulCcT0j", "yqQBUxeuga4", "tdwemEgGckz", "wDaqByzsz7p", "HgQ76w0VONy", "f87mpByuPub", "RHokurJAbjS", "w8SWd1reRm8", "IidAimBSgEf", "q8CUMlpFLtJ", "hR7B1ITcCMe", "qNxAIa6jvmm", "FZYvE3Jdf8L" ]; ```

Faulty detected yes/partial/no

```json { "module-2": [], "module-3": [ { "name": "Existence of national standard on mechanism on social accountability Medical Doctors", "id": "aEDrisPWh6i", "categoryCombo": { "name": "Key Occupations", "id": "CVlyw498LdE" } } ], "module-4": [], "module-6": [], "module-8-10": [ { "name": "Existence of institutional models for assessing and monitoring staffing needs for health services delivery", "id": "HgQ76w0VONy", "categoryCombo": { "name": "default", "id": "JzvGfLYkX17" } }, { "name": "Percentage of HWF at the primary health care level/facilities", "id": "LW954y3FqL8", "categoryCombo": { "name": "default", "id": "JzvGfLYkX17" } } ] } ```

Probable metadata bugs found

#### Module 3 - Existence of national standard on mechanism on social accountability Medical Doctors (aEDrisPWh6i) Instead of pointing to the yes/partial/no combo it is matching to Key Occupations combo. This is likely a typo when creating the form. The data entry form appears to record the dataValueSets correctly, but data visualization might not be working properly for this dataElement. #### Module 8-10 - Existence of institutional models for assessing and monitoring staffing needs for health services delivery (HgQ76w0VONy) It has no combo assigned (default). The data entry appears to record the dataValueSets correctly, but there might be side effects in data visualization.

Verification that all yes/partial/no are exclusive

* **Module 2:** 1/1 * **Module 3:** 61/63 (Two were missing: qZ9h4jv9cgO and fhe12emlYec) * **Module 4:** 1/1 * **Module 6:** 9/9 * **Module 8-10:** 15/14 (One was incorrectly tagged as yes/partial/no: LW954y3FqL8)

Edge cases that need to be treated in SQL mapping

- Module 3: qZ9h4jv9cgO, fhe12emlYec - Module 8-10: LW954y3FqL8

Script to find/delete invalid/non-exclusive data elements

```ts import axios from "axios"; import _ from "lodash"; import fs from "fs"; // CONFIGURATION FLAGS const BASE_URL = "http://user:pass!@taris.sferadev.com:9005"; const REMOVE_INVALID_FLAG = false; const REMOVE_DUPLICATES_FLAG = true; const ONLY_PRINT_INVALIDS = false; const ONLY_PRINT_REMOVED = false; const INCLUDE_OTHER_DATASETS = false; const SHOW_PERCENTAGE = true; const otherDataSets = [ { name: "NHWA Module 1", id: "Tu81BTLUuCT", }, { name: "NHWA Module 1 (Subnational)", id: "Humg4HbkmJg", }, { name: "NHWA Module 5", id: "cxfAcMbSZe1", }, { name: "NHWA Module 7", id: "ZRsZdd2AvAR", }, ]; const dataSets = [ ...(INCLUDE_OTHER_DATASETS ? otherDataSets : []), { name: "NHWA Module 2", id: "m5MiTPdlK17", }, { name: "NHWA Module 3", id: "pZ3XRBi9gYE", }, { name: "NHWA Module 4", id: "HtZb6Cg7TXo", }, { name: "NHWA Module 6", id: "WDyQKfAvY3V", }, { name: "NHWA Module 8-10", id: "p5z7F51v1ag", }, ]; const start = async () => { for (const [dataSetIndex, { id: dataSetId, name: dataSetName }] of dataSets.entries()) { const { organisationUnits } = (await axios.get(`${BASE_URL}/api/dataSets/${dataSetId}`, { params: { fields: "organisationUnits[id,name]" }, })).data; for (const [ organisationUnitIndex, { id: organisationUnitId, name: organisationUnitName }, ] of organisationUnits.entries()) { const { dataValues } = (await axios.get(`${BASE_URL}/api/dataValueSets`, { params: { orgUnit: organisationUnitId, dataSet: dataSetId, startDate: "1900-01-01", endDate: new Date().toISOString().slice(0, 10), }, })).data; const invalid = _.remove( dataValues, ({ categoryOptionCombo }) => !_.includes(["I93t0K7b1oN", "Xgr3PJxcWfJ", "Y7EAGQA1bfv"], categoryOptionCombo) ); const enabled = _.filter(dataValues, ["value", "true"]); const grouped = _.groupBy( enabled, ({ dataElement, period, orgUnit }) => `${orgUnit}-${period}-${dataElement}` ); const lastUpdated = _(grouped) .mapValues(array => _.maxBy(array, "lastUpdated")) .values() .value(); const toBeRemoved = _(grouped) .values() .flatten() .difference(lastUpdated) .value(); if (REMOVE_INVALID_FLAG && invalid.length > 0) fs.writeFileSync( `./elements-invalid-${dataSetId}-${organisationUnitId}.json`, JSON.stringify({ dataValues: invalid }, null, 4), "utf-8" ); if (REMOVE_DUPLICATES_FLAG && toBeRemoved.length > 0) fs.writeFileSync( `./elements-removed-${dataSetId}-${organisationUnitId}.json`, JSON.stringify({ dataValues: toBeRemoved }, null, 4), "utf-8" ); if (REMOVE_INVALID_FLAG && invalid.length > 0) await axios.post( `${BASE_URL}/api/dataValueSets`, { dataValues: invalid }, { params: { importStrategy: "DELETE", skipLastUpdated: true, skipAudit: true } } ); if (REMOVE_DUPLICATES_FLAG && toBeRemoved.length > 0) await axios.post( `${BASE_URL}/api/dataValueSets`, { dataValues: toBeRemoved }, { params: { importStrategy: "DELETE", skipLastUpdated: true, skipAudit: true } } ); if ( (!ONLY_PRINT_INVALIDS && !ONLY_PRINT_REMOVED) || (ONLY_PRINT_INVALIDS && invalid.length > 0) || (ONLY_PRINT_REMOVED && toBeRemoved.length > 0) ) { const percentage = Math.round( ((dataSetIndex * organisationUnitIndex + organisationUnitIndex) / (dataSets.length * organisationUnits.length)) * 100 ); console.log( dataSetName, organisationUnitName, `(${organisationUnitId})`, SHOW_PERCENTAGE ? `${percentage}%` : "" ); console.log(`Found ${invalid.length} invalid elements`); console.log(`Found ${toBeRemoved.length} elements to be removed`); console.log("-----------------------------------------------"); } } } }; start(); ```

Impact analysis of the bug

We have only reviewed modules affected by yes/partial/no: 2, 3, 4, 6, 8-10

Non-exclusive data values found in DEV (to be removed/cleaned)

NHWA Module 2 Burkina Faso (hmZE3mVAZFf)
Found 4 elements to be removed
-----------------------------------------------
NHWA Module 3 Burkina Faso (hmZE3mVAZFf)
Found 78 elements to be removed
-----------------------------------------------
NHWA Module 3 Central African Republic (ZcrjvJaYQb7)
Found 16 elements to be removed
-----------------------------------------------
NHWA Module 3 Democratic Republic of the Congo (EDHO4qOyY88)
Found 1 elements to be removed
-----------------------------------------------
NHWA Module 4 Burkina Faso (hmZE3mVAZFf)
Found 3 elements to be removed
-----------------------------------------------
NHWA Module 6 Burkina Faso (hmZE3mVAZFf)
Found 13 elements to be removed
-----------------------------------------------
NHWA Module 6 Central African Republic (ZcrjvJaYQb7)
Found 12 elements to be removed
-----------------------------------------------
NHWA Module 8-10 Burkina Faso (hmZE3mVAZFf)
Found 29 elements to be removed
-----------------------------------------------
NHWA Module 8-10 Central African Republic (ZcrjvJaYQb7)
Found 21 elements to be removed
-----------------------------------------------
Done in 150.35s.

Invalid data values found in DEV

Should we export them and remove from PROD?

NHWA Module 2 Republic of Angola (jFOZHDZpjPL)
Found 3 invalid elements
-----------------------------------------------
NHWA Module 2 People’s Democratic Republic of Algeria (mNa42CHbkO7)
Found 11 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Namibia (rtLnlu4GUI2)
Found 10 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of South Africa (lVPoUAKCdmU)
Found 4 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Rwanda (UlQiEogy3wG)
Found 25 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Madagascar (bcy4159FETR)
Found 24 invalid elements
-----------------------------------------------
NHWA Module 2 Union of the Comoros (RTdjvXHJdnH)
Found 16 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Uganda (SnYHrnchKjL)
Found 16 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Botswana (c82mDnhUQly)
Found 21 invalid elements
-----------------------------------------------
NHWA Module 2 Eritrea (vnbnnSZXGTv)
Found 16 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Mozambique (Gan3VYicAWe)
Found 128 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Zimbabwe (buSEeeViTo3)
Found 9 invalid elements
-----------------------------------------------
NHWA Module 2 Federal Democratic Republic of Ethiopia (zEYrsiNUGIo)
Found 22 invalid elements
-----------------------------------------------
NHWA Module 2 Malaysia (BmTVMvJHVBO)
Found 2 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Burundi (Xz7rnovuiOx)
Found 23 invalid elements
-----------------------------------------------
NHWA Module 2 United Republic of Tanzania (S52uSY3lb8V)
Found 12 invalid elements
-----------------------------------------------
NHWA Module 2 Burkina Faso (hmZE3mVAZFf)
Found 23 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Vanuatu (xqjIL1cGrl1)
Found 37 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Malawi (G037PAPU5dO)
Found 21 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Zambia (gb7vWtO7Wjp)
Found 19 invalid elements
-----------------------------------------------
NHWA Module 2 Kingdom of Thailand (vboedbUs1As)
Found 1 invalid elements
-----------------------------------------------
NHWA Module 2 Democratic Republic of the Congo (EDHO4qOyY88)
Found 17 invalid elements
-----------------------------------------------
NHWA Module 2 Republic of Kenya (HfVjCurKxh2)
Found 38 invalid elements
-----------------------------------------------
NHWA Module 3 People’s Democratic Republic of Algeria (mNa42CHbkO7)
Found 61 invalid elements
-----------------------------------------------
NHWA Module 3 Republic of Rwanda (UlQiEogy3wG)
Found 8 invalid elements
-----------------------------------------------
NHWA Module 3 Burkina Faso (hmZE3mVAZFf)
Found 9 invalid elements
-----------------------------------------------
NHWA Module 4 Islamic Republic of Iran (SsAjVE1S87E)
Found 2 invalid elements
-----------------------------------------------
NHWA Module 6 People’s Democratic Republic of Algeria (mNa42CHbkO7)
Found 21 invalid elements
-----------------------------------------------
NHWA Module 6 Republic of Mali (EubjsxqlA4d)
Found 1 invalid elements
-----------------------------------------------
NHWA Module 6 Burkina Faso (hmZE3mVAZFf)
Found 1 invalid elements
-----------------------------------------------
NHWA Module 6 Central African Republic (ZcrjvJaYQb7)
Found 1 invalid elements
-----------------------------------------------
NHWA Module 6 Federal Republic of Nigeria (I3NxIqG7bD4)
Found 1 invalid elements
-----------------------------------------------
NHWA Module 8-10 People’s Democratic Republic of Algeria (mNa42CHbkO7)
Found 14 invalid elements
-----------------------------------------------
NHWA Module 8-10 Burkina Faso (hmZE3mVAZFf)
Found 4 invalid elements
-----------------------------------------------
Done in 151.06s.

EyeSeeTea / WHO-packages

[HWF] Yes/Partial/No #3