Generate gentropy-based L2G evidence

d0choa commented 4 weeks ago

Data

The new GWAS credible set evidence will require the next fields:

The generation of this data needs to be implemented by taking the l2g_predictions and exploding the EFOs using the credible sets and study_index. For now, we will reproduce the previous L2G threshold of 0.05.

locus2geneId - Can be hashed from studyLocusId and targetId. New field in the json_schema
studyLocusId - New field in the json_schema
targetFromSourceId
diseaseFromSourceId
resourceScore L2G scoring will be used to score the evidence (as before)
datasourceId - to be named gwas_credible_sets

As discussed with @DSuveges, it would be good to eventually complete the evidence with more metadata to make it more readable using the pre-existing fields in the schema (e.g. studyId, pValue, etc.). This is only scoping the minimum set of data that MUST be there.

Platform ETL

To consume this data we will need to run the platform ETL (from evidence downwards). For now, l would run this evidence side-by-side with the current evidence, even though associations will be corrupted. Eventually we will drop the old evidence.

Suggested changes to ETL Reference.conf

{
      id: "gwas_credible_sets",
      datatype-id: "genetic_association"
      unique-fields: [
        "studyLocusId",
      ],
      score-expr: "resourceScore"
    }
}

Note: If I remember correctly targetId and diseaseId are unique-fields by default, so this is just what else defines unicity.

Evidence API

New columns that need to be added to the evidence endpoint:

locus2geneId - String -> NOT YET resolvable
studyLocusId - String -> Resolvable through credible sets API

A follow up action will be to populate the top L2G column in the credible sets using this data. But we can scope that once this data has been loaded.

@ireneisdoomed could you manually hand-craft this dataset in JSONL for @jdhayhurst? I think we have been using studyLocusId from 22.06 so perhaps it's the less problematic one to use for now.