Gentropy Airflow pipeline documentation - Githubissues

opentargets / issues

Issue tracker for Open Targets Platform and Open Targets Genetics Portal

https://platform.opentargets.org https://genetics.opentargets.org

Apache License 2.0

12 stars 2 forks source link

Gentropy Airflow pipeline documentation #3345

Open project-defiant opened 2 weeks ago

project-defiant commented 2 weeks ago

As a community user I want to be able to understand how the Gentropy ariflow pipeline works because I want to reproduce it with nextlfow.

Background

Background to this task is a result of a request send by community that stated:

level of confusion around the four steps involved in ingestion of the GWAS catalog data
- gwas_catalog_study_curation
- gwas_catalog_study_inclusion
- gwas_catalog_ingestion
- gwas_catalog_sumstat_preprocess
Investigate usage of nextflow with gentropy and check the possibility of swapping airflow with nextflow.

Tasks

[ ] Create screenshots from current airflow DAGs
[ ] Explain steps used to ingest the GWAS Catalog data

Acceptance tests

How do we know the task is complete?

Updated documentation around the GWAS Catalog ingestion process

project-defiant commented 1 week ago

The dags description:

gnomAD preprocess

gnomad_preprocess

Finngen harmonisation

finngen_harmonisation

eQTL preprocess

eqtl_preprocess

FInngen preprocess

finngen_preprocess

GWAS Curation update

gwas_curation_update

GWAS Catalog harmonisation

gwas_catalog_harmonisation

GWAS catalog preprocess

gwas_catalog_preprocess

Genetics ETL

genetics_etl