Load the latest version of the gentropy outputs

[d0choa]

This is a new iteration of the data, making #3567 obsolete:

❯ gsutil ls gs://ot_orchestration/releases/24.10_freeze2/
gs://ot_orchestration/releases/24.10_freeze2/locus_to_gene_gold_standard.json
gs://ot_orchestration/releases/24.10_freeze2/biosample_index/
gs://ot_orchestration/releases/24.10_freeze2/colocalisation/
gs://ot_orchestration/releases/24.10_freeze2/credible_set/
gs://ot_orchestration/releases/24.10_freeze2/gene_index/
gs://ot_orchestration/releases/24.10_freeze2/invalid_credible_set/
gs://ot_orchestration/releases/24.10_freeze2/invalid_study_index/
gs://ot_orchestration/releases/24.10_freeze2/locus_to_gene_feature_matrix/
gs://ot_orchestration/releases/24.10_freeze2/locus_to_gene_model/
gs://ot_orchestration/releases/24.10_freeze2/locus_to_gene_predictions/
gs://ot_orchestration/releases/24.10_freeze2/manifests/
gs://ot_orchestration/releases/24.10_freeze2/study_index/
gs://ot_orchestration/releases/24.10_freeze2/variant_index/
gs://ot_orchestration/releases/24.10_freeze2/variants/

A frozen description of all schemas can be found here.

The changes in the schemas, added to the ones described in #3567 are:

• log2h4h3 has been dropped in the colocalisation schema

The full changelog of this dataset compared to 24.10_freeze1:

• Sumstats PICS based on all harmonised studies 77k (69k non-empty ones)
• Major ancestry-based PICS (top-hits + sum stats)
• Duplicated top-hits removed due to duplicated associations in GWAS catalog file
• Lead stats included in locus column of PICS credible sets
• Bug failing to sort VCFs killing half of the batch jobs in variant index fixed
• Variant index partitioning revised. 200 partitions without using partitionBy
• UKBB PPP finemapping re-run (still GnomAD-based)
• UKBB PPP should no longer be dropped at validation stage due to lack of biosampleId
• log2h4h3 dropped from the coloc schema
• coloc priors now configured

A probably incomplete list of things that we know are pending in this dataset from a semantic perspective:

• Some finngen credible sets contain duplicated variants inside the locus column
• A fraction of QTLs (incl. iPS cells) are dropped due to EFO-based biosamples
• GWAS catalog study filtering based on curation + qc_sumstats
• hasSummaryStats field in the study index to be properly populated
• L2G evidence + any required adjustments
• Additional credible set validation (e.g. confirming PP sum 0.99-1)
• New GWAS Catalog SuSiE fine-mapping (pan-UKB LD reference, etc.)

This pipeline run was based on the gentropy szsz-nonmandatory-params-to-coloc-step (due to some adjustments required on the coloc priors) and do_etl_quick_iterations in the orchestration package. I will try to make the changes to dev in both repository as it gave us an almost perfect run. (@ireneisdoomed, it crashed feature matrix because of the broadcasting issue. I disabled auto-broadcast through configuration similar to train/predict and it worked well)

[d0choa]

This data has been superseded by : gs://ot_orchestration/releases/24.10_freeze3/ (generated by @project-defiant)

Changelog compared to 24.10_freeze2:

• fix on VM heap size causing incomplete variant_index - variants_to_vcf was failing due to heap space

A probably incomplete list of things that we know are pending in this dataset from a semantic perspective:

Some finngen and eQTL catalogue credible sets contain duplicated variants inside the locus column PICS results have an issue in which variantId is not shown inside the locus object A fraction of QTLs (incl. iPS cells) are dropped due to EFO-based biosamples GWAS catalog study filtering based on curation + qc_sumstats hasSummaryStats field in the study index to be properly populated L2G evidence + any required adjustments Additional credible set validation (e.g. confirming PP sum 0.99-1) New GWAS Catalog SuSiE fine-mapping (pan-UKB LD reference, etc.)

@DSuveges will report some metrics to make sure data is within the expected ranges

[project-defiant]

@d0choa https://github.com/opentargets/orchestration/pull/41#event-14696106544 <- configuration update for the last run

[prashantuniyal02]

24.10_freeze4

> gsutil ls gs://ot_orchestration/releases/24.10_freeze4/
gs://ot_orchestration/releases/24.10_freeze4/locus_to_gene_gold_standard.json
gs://ot_orchestration/releases/24.10_freeze4/biosample_index/
gs://ot_orchestration/releases/24.10_freeze4/colocalisation/
gs://ot_orchestration/releases/24.10_freeze4/credible_set/
gs://ot_orchestration/releases/24.10_freeze4/gene_index/
gs://ot_orchestration/releases/24.10_freeze4/invalid_credible_set/
gs://ot_orchestration/releases/24.10_freeze4/invalid_study_index/
gs://ot_orchestration/releases/24.10_freeze4/locus_to_gene_feature_matrix/
gs://ot_orchestration/releases/24.10_freeze4/locus_to_gene_model/
gs://ot_orchestration/releases/24.10_freeze4/locus_to_gene_predictions/
gs://ot_orchestration/releases/24.10_freeze4/manifests/
gs://ot_orchestration/releases/24.10_freeze4/study_index/
gs://ot_orchestration/releases/24.10_freeze4/variant_index/
gs://ot_orchestration/releases/24.10_freeze4/variants/

[d0choa]

Changelog for 24.10_freeze4:

• Number of colocalising variants - now reflects intersect
• eQTL catalog duplications - Re-run
• Finngen duplications
• rescued eQTL catalog studies due to richer Biosample index
• Finngen study id rescues FINNGEN_R11
• PICS credible set missing variant identifier fixed
• New PPP finemapping using pan-UKBB LD matrixes
• Sumstats PICS results should be now discarded due to overlapping SuSiE
• Confidence of eQTL catalog + Finngen should be now equivalent to 4 stars

All the changes above should not structurally change the data, but they will fix some issues in the FE. I mentioned to @jdhayhurst that we have made a schema change in studyIndex regarding the sum stats QC columns. I can't remember if we picked up these changes for this data release but if you experience issues with study index this is why.

Known issues as 22nd of October:

• GWAS catalog study filtering based on curation + qc_sumstats. The current data release is a superset of the credible sets in the final version
• Pan-UKBB finemapping of GWAS catalog still missing
• L2G evidence still missing from release data
• FinnGen and eQTL catalogue finemappingMethod is misspelt SuSie instead of SuSiE

[project-defiant]

@jdhayhurst The changes in the study index are refering to the sumstatQCValues key:value map we discussed.

[addramir]

Is it finsihed?

[prashantuniyal02]

30/10: Will wait for 24.10_freeze6 for BE to load the data till Friday. If not, load freeze5

[project-defiant]

Changelog for 24.10_freeze5

• Clean credible sets from loaded pre-validation with credible_set_qc step performed on them
  • ukb_ppp_eur - 30584 rows
  • gwas_catalog_sumstats_susie - 263732 rows
• Usage of gwas_catalog_pics studyIndex for validation of gwas catalog studies
• ABNORMAL_PIPS flag in the credible sets qc flags
• Disease/target evidence in JSON format is generated after L2G prediction.
• gwas_catalog_top_hits credible sets now have variantId correctly mapped to studyId
• L2GFeatureMatrix has 4 more columns: 'credibleSetConfidence', 'geneCount500kb', 'isProteinCoding', 'proteinGeneCount500kb'