jwestw
commented
3 years ago @LucyGwilliamAdmin I have been thinking about potential approaches to this. Some notes on what I think might work:
- For any step that needs manual intervention, there are two options, either 1) a fully manual step which would involve working on spreadsheets, saving the result, importing it into the next stage of the script, OR 2) a semi-manual step, which involves user interaction with an interface, and the script helping the user to select the correct options to be carried through to the next stage of the script.
- If we are going for option 1 (which is currently the case with the column matching step ++), then the user could change the file-name after they have made their choices. e.g. "output_stage_1.csv" could be changed to "input_stage_3.csv". That would prevent overwriting, next time the script is run, as the script would write out to "output_stage_1.csv", leaving the user's changes preserved.
- If we are going for option 2 (which is currently the case with the value/code matching step), then the script could output a file. Usually I would prefer preserving the Python objects as Pickle or feather, but we might want to have them human readable so they can be inspected outside of the script, so csv might be our best option here.
- We could allow the user to inspect the data within the interface, so they know what they are working with.
- Automatically outputted csv (or other) files could be dated, which would make the file names unique. We could develop an option where the user has the option of either picking up progress from the most recent file, another older file, or starting over. Not quite sure how we let the user interface with and change existing data...needs some more thought.
++ Side note: in terms of accuracy, reproducibility and user-experience, I would prefer that we converted the manual column matching step to a semi-manual step.
You asked me to make time estimates on this but it depends what approach we decide on.
When new indicators or values are introduced the script should deal with this and ask the user for updates to the manual mappings, rather than overwriting what is already there.
I will break this problem down and make a time estimate.