Simplify store_data() arguments, requirements on data passed

[noleti]

This issue is to discuss next steps assuming that case-counts folder structure is simplified as suggested in https://github.com/neherlab/covid19_scenarios_data/issues/63, and direct .json generation is dropped do allow for manual verification/diffing of the .tsv data.

Proposed signature of store_data():

def store_data(data, source, cols=[]):

Lets assume the main data structure passed by the parser is called data and has the format {'USA': [{'time': '2020-01-20', 'cases': 20,...},..,{'time': '2020-03-20', 'cases': 200,...}]} or {'USA': [['2020-01-20', 20,...],..,['2020-03-20', 200,...]]} I would like to propose the following:

• parsers should now explicitly be responsible to set country-level keys in data: country level is would just be name of country as found in country_codes.csv. State-level it needs to be the three-letter country code from country_codes.csv, a hyphen, and then the state name (e.g., USA-New York. We would need to update existing parsers to do so. ecdc and cds will not have to be updated. For others, the existing exceptions dict should tell you which keys are country level, all others will be state-level and need to be prepended by the three letter country code.
• source is the string identifying the parser, matching sources.json. It will also be used to name the folder in case-counts for the .tsv files
• cols will still be required to be able to parse data in case that is a dict of lists of lists (done in some parsers), and we don't want to rely on the parser passing data in the correct order

store_data() could then be simplified a lot. I would suggest:

• If we want to drop direct .json generation, then we can just check if data is a dict of lists of dicts. if yes, we convert to dict of list of lists using dict_to_list(regions, default_cols). Either way, we then call store_tsv(). That function can also be simplified to get rid of the world.tsv and exception handling (would not be needed, as state-level keys would have appropriate names). I would still recommend sanitization of API-provided strings when using them for filenames. The files would then be saved to BASE_PATH/{source}/{country-or-state-name}.tsv
• store_json() and merge_cases(), compare_day() can likely be reused for the later parsing of .tsv into json, so I would recommend not just deleting and forgetting about them.

@tryggvigy tagging you as you explicitly said you would like to do this. Hope this helps. I can also do it, let me know if I should.

[tryggvigy]

@noleti thanks for involving me, I appreciate it and all the context you provide helps me get into the codebase. I think it's best if you do it. I won't be able to contribute as much during weekdays and I don't want to bite off too much now. I'll keep an eye out for your PR to see how you've done it. I bet I'll learn a lot from that.

[noleti]

resolved in https://github.com/neherlab/covid19_scenarios_data/pull/68