Closed toneillbroad closed 3 weeks ago
We need to think a little bit about what do to when a "fix" release is published by ClinVar. For example sometimes they publish a release again the next day to censor PHI or fix data published the day before. (I think I've seen them publish a fix release within the same day).
I'm not sure if the FTP watcher will detect the fix file as being a "new" file to be processed.
I think the "fix" release even if published the next day will have the ReleaseDate and file name being the same as the release file it is fixing. But the modified date on the file metadata will be the new timestamp.
Closing due to this being re-issued as this epic: https://app.zenhub.com/workspaces/genegraphdxclinvar-60340fb9898dae001107e94e/issues/gh/clingen-data-model/clinvar-curation-input-tool/85
The clinvar VCV and RCV ingests will be invoked by their respective FTP watcher processes with the release date information derived from their respective filenames.
For example: ClinVarVariationRelease_2024-0611.xml.gz -> Release Date: "2024-06-11" ClinVarRCVRelease_2024-0610.xml.gz -> Release Date: "2024-06-10"
The implications of this is that the parsed files that are the basis of the external BQ tables are generated in a GCP bucket with the release date as part of the name.
ClinVarVariationRelease_2024-0611.xml.gz -> gs://clinvar-ingest/executions/clinvar_2024_06_11_v1_0_0_alpha
With the VCV and RCV files above having different release dates we need to ensure that the parsed files are imported into the proper BQ dataset name as it constains the date as part of the dataset name:
ClinVarVariationRelease_2024-0611.xml.gz -> Dataset ID: clingen-dev.clinvar_2024_06_11_v1_0_0_alpha
A fundamental tenet of the RCV and VCV ingests are that they are not dependent on one of the files always coming first and the other second - but rather they should process in the order in which they appear on the ClinVar FTP site.
Therefore, the strategy for determining the proper BQ dataset to ingest the parsed files into is as follows:
Perhaps we should consider appending "_vcv" or "_rcv" to the gs://clinvar-ingest/executions parsed data files: gs://clinvar-ingest/executions/clinvar_2024_06_24_v1_0_0_alpha -> gs://clinvar-ingest/executions/clinvar_2024_06_24_v1_0_0_alpha_vcv