Gentropy Airflow pipeline documentation

[project-defiant]

As a community user I want to be able to understand how the Gentropy ariflow pipeline works because I want to reproduce it with nextlfow.

Background

Background to this task is a result of a request send by community that stated:

• level of confusion around the four steps involved in ingestion of the GWAS catalog data
  • gwas_catalog_study_curation
  • gwas_catalog_study_inclusion
  • gwas_catalog_ingestion
  • gwas_catalog_sumstat_preprocess
• Investigate usage of nextflow with gentropy and check the possibility of swapping airflow with nextflow.

Tasks

• [x] Create screenshots from current airflow DAGs
• [x] Explain steps used to ingest the GWAS Catalog data

Acceptance tests

How do we know the task is complete?

1. Updated documentation around the GWAS Catalog ingestion process

[project-defiant]

The dags description:

gnomAD preprocess

Finngen harmonisation

eQTL preprocess

FInngen preprocess

GWAS Curation update

GWAS Catalog harmonisation

GWAS catalog preprocess

Genetics ETL

[project-defiant]

Substantial changes in the way how we process GWAS Catalog are documented in the orchestration repository in https://github.com/opentargets/orchestration/tree/dev/docs

Current documentation effords:

• [x] GWAS Catalog sumstat susie
• [x] GWAS Catalog sumstat pics
• [x] GWAS Catalog top hits
• [x] UKB PPP EUR
• [x] FInngen r11
• [x] Credible_set_qc
• [ ] GWAS Catalog sumstat harmonisation
• [ ] Genetics ETL