Attempting to generate multiple Clinvar summary tables, each representing different dates, failed to produce accurate results
Proposed Changes
AFAIK the table generation fault is not a bug, it's an issue with multiple version of the script running in the same environment at the same time. Specifically, the JSON written to temp has a completely generic file name, so multiple concurrent versions will read/write this same file
This change introduces a couple of changes to keep parallel run results separate
the summary and variant data copied from NCBI are persisted with a date in the file path to make it simpler to determine when a file was originally copied. This data is pre-processed, so this date only has to represent the time it was copied, not what was done to process it later.
the temporary JSON file contains the target date as a String - other than the blacklisted sites etc (which are cohort-dependent), the date will be the main difference between parallel runs, so this should keep things separate and identifiable
A few more logging statements were added
A date argument was switched from DD-MM-YYYY to YYYY-MM-DD to mirror all other date formats used in this project
Fixes
Proposed Changes
Checklist