Write the file stager/filter for the harmonisation pipeline

[jdhayhurst]

The harmonisation pipeline will process anything in a "ready to harmonised" directory. We need a script to deposit files into this directory assuming they are eligible.

Reduce “glue” by adding this to the existing Sumstats (ftp) release script, which already needs to identify and release newly submitted files.

• Find new/modified files since last run -- Release on FTP
• Check eligibility for harmonisation -- Currently: query to API that it is part of publication -- Future: read the metadata  when sumstats go from unpublished to published the metadata (file) will need updating
• Add to harmonisation queue
• Log timestamp for current run

By adding into the existing “nighlty sumstast ftp sync” script, we increase the complexity of what that script does but we remove another gluey script. We can remove another glue script by using the publishing directives of nextflow.

more info: https://docs.google.com/document/d/1b1g9PIUH6B688_aqBIaZulOtgCEnEXNcirJ1vUUmJq8/edit#

[jdhayhurst]

rather than a black/white list, we can actually use the metadata for the files to store the validation status - if valid, harmonised; else don't.

[jdhayhurst]

• read last run timestamp from file
• find newer sumstats files since timestamp
• store current timestamp in last run file
• check if files are eligible using yaml: valid = true, published = true
• cp eligible files to "ready to harmonise" dir

Note: if files are "flipped" into a published state, we need to a) update this in the YAML + b) touch the sumstats file, so that they will be picked up by the above.

[jdhayhurst]

To reduce the amount of "glue" we can add this into the ftp sync process - https://github.com/EBISPOT/gwas-utils/ftpSummaryStatsScript/ftp_sync.py