Explore impact of ambigious usage of Ensembl and Gnomad based coordinates for indels

[DSuveges]

It has been noticed in the past (upon mapping GWAS Catalog curated associations), that coordinates of indel in Ensembl and GnomAD different by 1 position. (more info here: https://www.biostars.org/p/84686/). This affects indels only. This discrepancy causes that indels from GWAS Catalog could not be resolved against GnomAD variant reference, leading to there's no indels in the production genetics portal.

Tasks:

• [ ] Find out how the GWAS Catalog summary stats look like?
• [ ] Check indels' position and alleleles, are the alleles contain '-'?
• [ ] Is the marked position match with GnomAD position?

Based on what we find we need to find out how to unify the indel representation in gentropy.

[DSuveges]

Based on a discussion with @addramir and @d0choa, we agreed that changing the coordinates of the variant annotation table is too disruptive with a wide series of negative consequences in variant data harmonisation and ingestion. So it has been decided that we revert:

• [x] The variant annotation coordinates should remain unchanged.
• [x] To make sure we can normalise curated GWAS Catalog associations to Gnomad, we have to termoporarlity update coordinates, and afterwards these coordiates should be reverted to their original form.

The datasets that need to be re-generated:

• [x] GnomAD4 based variant annotation
• [x] GWAS Catalog curation ingestion

[DSuveges]

QC Variant annotation

Link to QC notebook: here

Conclusions:

• The reference for variant annotation in the configuration point to a dataset based on GnomAD3.
• The gnomad3VariantId column is dropped. Good.
• Could verify that the same indel variant in the old and the new show different in position:

New data

-RECORD 0---------------------------
 variantId       | 10-41776411-C-CT 
 chromosome      | 10               
 position        | 41776411         
 referenceAllele | C                
 alternateAllele | CT  

Old data:

-RECORD 0----------------------------
 variantId        | 10_41776412_C_CT 
 chromosome       | 10               
 position         | 41776412         
 referenceAllele  | C                
 alternateAllele  | CT               
 gnomad3VariantId | 10-41776411-C-CT 

QC LD-index

• The size of the LD index doesn't change.
• The coordinates of indels in the LD index is changed:

Old ld-index:

+-----------------+
|variantId        |
+-----------------+
|6_100124716_CA_C |
|6_100124716_C_CA |
|6_100124715_C_A  |
|6_100124716_C_CAA|
+-----------------+

New ld-index:

+-----------------+
|variantId        |
+-----------------+
|6_100124715_C_CA |
|6_100124715_C_CAA|
|6_100124715_C_A  |
|6_100124715_CA_C |
+-----------------+

[d0choa]

It might be my misunderstanding but mentioning just in case. The variantId in the new data pasted above has hyphens instead of underscores. That would not play well with everything else.

[DSuveges]

It might be my misunderstanding but mentioning just in case. The variantId in the new data pasted above has hyphens instead of underscores. That would not play well with everything else.

Yes, you are right! I just renamed the column from gnomad3VariantId to variantId. Applying the fix. Done.

[DSuveges]

Fix applied, code pushed to branch. DAG is re-run, saved here: gs://genetics_etl_python_playground/output/python_etl/parquet/XX.XX/variant_annotation

Sample:

-RECORD 0----------------------------------
 variantId          | 10_41776397_C_G      
 chromosome         | 10                   
 position           | 41776397             
 chromosomeB37      | 10                   
 positionB37        | 42463812             
 referenceAllele    | C                    
 alternateAllele    | G                    
 rsIds              | [rs1427850406]       
 alleleType         | snv                  
 alleleFrequencies  | [{afr_adj, 0.0}, ... 
 vep                | {intergenic_varia... 
 inSilicoPredictors | {{1.028, -0.05497... 
only showing top 1 row