Staging data expectations Sep 30th

[d0choa]

We want to create a version of the staging data that complies with the following rules for a gentropy pipeline run.

This is a data-patching exercise with the intention to produce a clean run of the pipeline and identify further actions in the implementation of the logic:

The next are all the staging inputs:

study_index_path:
        - gs://gwas_catalog_data/study_index
        - gs://eqtl_catalogue_data/study_index
        - gs://ukb_ppp_eur_data/study_index
        - gs://finngen_data/r11/study_index
 study_locus_path:
        - gs://gwas_catalog_data/credible_set_datasets/gwas_catalog_PICSed_curated_associations
        - gs://gwas_catalog_data/credible_set_datasets/gwas_catalog_PICSed_summary_statistics
        - gs://gwas_catalog_data/credible_set_datasets/gwas_catalog_susie_summary_statistics
        - gs://eqtl_catalogue_data/credible_set_datasets/eqtl_catalogue_susie/
        - gs://ukb_ppp_eur_data/credible_set_datasets/ukb_ppp_eur_susie
        - gs://finngen_data/r11/credible_set_datasets/finngen_r11_susie

Requirements:

• All StudyLocusId need to be in String format
• All gwas_catalog_PICSed_curated_associations need to have in the qualityControls column the StudyLocusQualityCheck.TOP_HIT
• All gwas_catalog_susie_summary_statistics, eqtl_catalogue_susie, finngen_r11_susie and ukb_ppp_eur_susie need to have the columns locusStart and locusEnd populated derived from the region column
• All gwas_catalog_susie_summary_statistics and ukb_ppp_eur_susie need a new flag to be added StudyLocusQualityCheck.OUT_OF_SAMPLE_LD

@DSuveges, @addramir I don't think I'm missing anything but have a quick read through

The moment this data is ready we will want to rerun the DAG (@project-defiant)

[vivienho]

The data has been patched!