new workflow to auto-generate OptionsMap

aleksa-krolls commented 2 months ago

Request

Then, every time the client updates the options map, I have to manually generate this gist. To automate this process and save us time, can you please help me create a new workflow to auto-generate this OptionsMap?

Input

A csv file that I'll upload to GoogleSheets. It would be nice if in the job I could specify the name of the file I want to pull.

Example file (you can assume the csv files will always have the same column names: https://drive.google.com/file/d/1OTXLjnFHhR4Nety9mRE_LbtC_r2ZD7w-/view?usp=drive_link

Output

A json file that looks like this gist: https://gist.github.com/aleksa-krolls/2f309595110239cfcc8251017cf11592

Draft mapping:

value.display - Answers: csv.Answers
value.uuid - External ID": csv.External ID
DHIS2 DE full name: DHIS2 DE full name
DHIS2 DE UID: csv.DHIS2 DE UID
OptionSet name: csv.OptionSet name
DHIS2 Option Set UID: csv.DHIS2 Option Set name
DHIS2 Option name: csv.DHIS2 Option name
DHIS2 Option UID: csv.DHIS2 Option UID
DHIS2 Option Code: csv.DHIS2 Option code

mtuchi commented 2 months ago

Quick Feedback

The file in google drive must be a googlesheet file so that we can use googlesheet adaptor to fetch the values

After we generate the mapping, what should we do with empt data that is being mapped. For example

{
  "DHIS2 DE UID": "",
  "DHIS2 DE full name": "",
  "DHIS2 Option Code": "",
  "DHIS2 Option Set UID": "",
  "DHIS2 Option UID": "",
  "DHIS2 Option name": "",
  "OptionSet name": "Vulnerability criteria - The child is",
  "value.display - Answers": "Adopted",
  "value.uuid - External ID": ""
},

aleksa-krolls commented 2 months ago

@mtuchi cool, then go ahead and convert to GoogleSheet --> and I'm okay with following the process in the future.

And if DHIS2 DE full name OR value.uuid - External ID are not defined ( "" or NA in the input)... then filter this out and do NOT include these rows in OptionMap.

mtuchi commented 2 months ago

@aleksa-krolls should the filter be OR or AND ? Because if one of the value is true between DHIS2 DE full name and value.uuid - External ID is true then we will still have "NA" or "" in the values that is false

Eg: value.uuid - External ID has a valid value but DHIS2 DE full name has invalid value 👇🏽

{
      "DHIS2 DE UID": "NA",
      "DHIS2 DE full name": "NA",
      "DHIS2 Option Code": "",
      "DHIS2 Option Set UID": "NA",
      "DHIS2 Option UID": "NA",
      "DHIS2 Option name": "NA",
      "OptionSet name": "Sex - All",
      "value.display - Answers": "Male",
      "value.uuid - External ID": "625baaf5-ba75-4c23-bdef-283fd47c34db"
    },

Eg: DHIS2 DE full name has valid value but value.uuid - External ID has invalid value

  {
      "DHIS2 DE UID": "Xvzc9e0JJmp",
      "DHIS2 DE full name": "Nationality",
      "DHIS2 Option Code": "",
      "DHIS2 Option Set UID": "GL - Nationality",
      "DHIS2 Option UID": "CcpigvKey7O",
      "DHIS2 Option name": "China",
      "OptionSet name": "Country",
      "value.display - Answers": "China",
      "value.uuid - External ID": ""
    },

aleksa-krolls commented 2 months ago

@mtuchi pls also filter out rows if "DHIS2 DE full name": "NA" OR if value.uuid - External ID": ""

All of these should be OR scenarios... so only if DHIS2 DE full name and value.uuid - External ID have legit values (not empty strings or NA) should we include in the optionMap

mtuchi commented 2 months ago

@aleksa-krolls noted, I have update the workflow accordingly, See output in this latest workflow run

OpenFn / openfn-lime-pilot