Resolve data issues with new `target` ETL and API endpoint

[andrewhercules]

This ticket captures preliminary findings after reviewing the development API with the new target ETL data.

19 July 2021: ticket still a work-in-progress as review ongoing

Mouse Phenotypes

Issues with MP should be compiled on a specific MP ticket.

Pathways

• [x] No field is available to query; no data is returned by the API

Comparative Genomics

• [x] The values used in the "Query %id" (queryPercentageIdentity) and "Target %id" (targetPercentageIdentity) are different compared to the UI data table that relies on a direct call to the Ensembl API (e.g. CFTR)
• [x] Some species are missing (e.g. ESR1 is missing “Macaque”, “Pig”, “Mouse”, “Guinea pig”, “Rabbit”, “Fly”, “Frog”)
• [x] Multiple entries for the same species are missing (e.g. TNF is missing second "Rat" entry)
• [x] Additional species are included (e.g. TNF has entry for “Tropical clawed frog”)

Protein Information

• Some locations have semicolons (e.g. LRRK2 and "Mitochondrion outer membrane ; Peripheral membrane protein") and commas (e.g. APP and "Cell projection, growth cone") - is this expected or do these strings represent separate locations?

Target Safety

• No assay format, assay description, or assay type data for Tox21 or eTOX (e.g. ABCG2, SLC10A2, F3)
• Tox21 tissue mapping should be reviewed (e.g. ABCG2 has tissue.label values of "placenta", "Intestine", "breast", while the while tissue.modelName is hepatocyte)
• Unable to test Known Safety effects data (e.g. for ACHE - ENSG00000087085)

TEPs

• Data is okay
• No changes needed in front-end

Cancer Hallmarks

• Data is okay
• Changes required in front-end - see #1640

Target tractability

• [x] Data is okay (checked AR, ESR1, TNF for various use cases)
• [x] Changes required in front-end - see #1644
• [x] #1672

Cancer Biomarkers

• Changes required in front-end-end - see #1645
• [ ] Resolve data duplication - possibly caused by "exploding" a single row found in the underlying dataset

Gene Ontology

• [ ] No data returned by GraphQL API for ESR1 or TNF or MTOR
• [ ] Internal server error when searching ACHE (could be the safety data?)
  • Yes, this is caused by bad safety data. I'll recheck it once that is fixed to make sure there isn't another underlying issue.

Chemical Probes

• [x] No data returned by API

Baseline expression

• [ ] No front-end changes expected - the API response structure still matches the original query used by the Platform

Target classes

• [x] Modify ETL output so either all targets have an empty array or empty arrays are null.

[JarrodBaker]

Thanks @andrewhercules, I've moved the mouse phenotype section to a new ticket as a) it's a separate index (internally), and b) those issues existed on the current dataset. I think that @d0choa had mentioned that we need to rewrite MP, so that can maybe a starting point for those.

Please keep adding issues here and I'll triage as they come in. :)

[JarrodBaker]

@andrewhercules pathway was removed from the schema and is no longer available. There was a comment that we could potentially get this from 'reactome pathways' if necessary (comment on target refactor spreadsheet iteration1, but it hasn't been followed up as yet. I think the broad plan was to introduce a new 'geneSets' index of some sort using reactome data as a base. @d0choa will have a better idea.

[d0choa]

my memory might fail but I think the Reactome pathways (R-HSA-XXX) in the new implementation were coming as xrefs directly from Ensembl? Could you check that field?

Resolving the ids into labels was done by @mkarmona for the facets.

[andrewhercules]

my memory might fail but I think the Reactome pathways (R-HSA-XXX) in the new implementation were coming as xrefs directly from Ensembl? Could you check that field?

Resolving the ids into labels was done by @mkarmona for the facets.

I have checked the dbXrefs field and it only returns the Pathway ID and source. Currently, we display the pathway name and top-level parent pathway.

[JarrodBaker]

Regarding comparative genomics:

• the values of query and target id percentages being different from the Ensembl API: this seems to be caused by the new target index being generated with an older Ensembl release. I've redeployed the index and the results between Ensembl and the API appear to align.
• I've updated the whitelist of species in the configuration which appears to have resolved some of the the missing species (along with using up-to-date data).
• Inclusion of additional species:
  • 'tropical clawed frog' is the common name for xenopus_tropicalis (taxonomy id 8364) used by Ensembl. Do we want to use the Ensembl name @andrewhercules / @d0choa or map that to 'frog'. If we want to change it we can do it in either the front or back end. What is 'tropical clawed frog' in the API is 'frog' in the current FE.
• Some species can be expected to disappear: for instance the Fly homologue on ESR1 is removed in the ETL as we filter for 'is_high_confidence' which removes that entry. As far as I can tell from quickly looking at the FE code we are just requesting all the homologues for an ENSG ID and doing no significance filtering on them.

I think we're still missing a couple of entries that we should have though, so I'll keep digging.

[JarrodBaker]

Ticking off the missing 'rat' entry on TNF as the second entry is low confidence so we're excluding it intentionally:

+---------------+-----------------+-----------------------+------------------+
| gene_stable_id| homology_species|homology_gene_stable_id|is_high_confidence|
+---------------+-----------------+-----------------------+------------------+
|ENSG00000232810|rattus_norvegicus|     ENSRNOG00000055156|                 0|
|ENSG00000232810|rattus_norvegicus|     ENSRNOG00000000837|                 1|
+---------------+-----------------+-----------------------+------------------+

Regarding ESR1:

• fly is missing because of low confidence
• the mouse entry seems to be missing from the raw data. When I look for mouse homologues on ESR1 I get

  +--------------------+
  |    homology_species|
  +--------------------+
  |          mus_caroli|
  |mustela_putorius_...|
  |         mus_spretus|
  |          mus_pahari|
  |      mus_spicilegus|
  +--------------------+

  whereas we're only looking for mus_musculus as specified in opentargets/platform#1047 (10090 is mouse). @d0choa is this something that we need to investigate further or do we assume the new data is actually the data we want?

[JarrodBaker]

Regarding protein information (subcellular location), taking the example of APP (ENSG00000142192)

+---------------+---------------------------------------------------+
|id             |location                                           |
+---------------+---------------------------------------------------+
|ENSG00000142192|Cell membrane ; Single-pass type I membrane protein|
|ENSG00000142192|Cell projection, growth cone                       |
|ENSG00000142192|Cytoplasm                                          |
|ENSG00000142192|Cytoplasmic vesicle                                |
|ENSG00000142192|Early endosome                                     |
|ENSG00000142192|Golgi apparatus                                    |
|ENSG00000142192|Membrane ; Single-pass type I membrane protein     |
|ENSG00000142192|Membrane, clathrin-coated pit                      |
|ENSG00000142192|Perikaryon                                         |
|ENSG00000142192|Vesicles                                           |
|ENSG00000142192|[Amyloid-beta protein 42]: Cell surface            |
|ENSG00000142192|[Gamma-secretase C-terminal fragment 59]: Nucleus  |
|ENSG00000142192|[Soluble APP-beta]: Secreted                       |
+---------------+---------------------------------------------------+

All of these entries come from Uniprot flat files which we parse in the ETL. The documentation says that:

• what is before a colon is a molecule;
• entries that contain comma separated values are single locations;
• semi-colons define a location hierarchy location ; topology ; orientation

This is formally defined as:

The format of SUBCELLULAR LOCATION is:

           CC   -!- SUBCELLULAR LOCATION:(( Molecule:)?( Location\.)+)?( Note=Free_text( Flag)?\.)?

Where:

    Molecule: Isoform, chain or peptide name
    Location = Subcellular_location( Flag)?(; Topology( Flag)?)?(; Orientation( Flag)?)?
        Subcellular_location: SL-line of subcell.txt ID-record
        Topology: SL-line of subcell.txt IT-record
        Orientation: SL-line of subcell.txt IO-record

In the current Platform:

• entries separated by commas are listed as separate entries
• any information after a semi-colon is discarded
• the information before a colon is discarded.

Uniprot:

• only lists the last part of the comma separated values (eg Membrane, clathrin-coated pit become clathrin-coated pit
• retains the information after the semi-colon
• display entries which include a colon separately, as they relate to other proteins.

In short, we've got a bit of a mix of both the current Platform and Uniprot. Any thoughts @andrewhercules and @d0choa on which way we want to go?

[JarrodBaker]

ChemicalProbes is yet to be implemented, see (opentargets/platform#1389). It's with @ireneisdoomed at present. Once she's back from leave she can provide an update.

[JarrodBaker]

Regarding Protein Information: the schema has changed so that evidence is a simple string. I've tried adding in eco codes as described in opentargets/platform#1037 but they are quite sparse. I'll revisit it again later. Unless I hear otherwise I'll assume that they are still a low priority (as mentioned in the linked ticket).

If nothing else the endpoint should be working!

[JarrodBaker]

I've temporarily removed targetSafety from the API as it's broken and unstable. This way other errors that are found will be more likely to be genuine errors.

[tskir]

Changes required to accommodate mousePhenotypes schema updates (https://github.com/opentargets/platform/issues/1471)

Comparison between the old and new schema is available in this spreadsheet (first four columns). Summary:

• [ ] The new schema fill be flat, compared to the previous one which was nested a few levels deep.
• [ ] All existing fields will be renamed to follow the camel case convention and also to align with the corresponding evidence object where possible.
• [ ] The contents of the existing fields will not change, except for fixing the PMID bug (where currently pipe symbol is used instead of an array) described in https://github.com/opentargets/platform/issues/1642.
• [ ] Two new fields are added: targetInModelEnsemblId and biologicalModelId.

Also CCing @ireneisdoomed @DSuveges to keep them in the loop.

[ireneisdoomed]

Cancer Biomarkers Updates

This dataset describes association between target and disease when a cancer biomarker is found.

Because of this particularity of the presence of the biomarker, it was initially scoped to be part of the target annotations and not part of the evidence.

We have modelled and parsed the table so that the cancer biomarkers are a new data source of evidence for the upcoming release (PR #89)

The schema that this source follows is tracked here: https://docs.google.com/spreadsheets/d/1Mowq7KsGTMtEg3wZpJBNK_UbawHKJeM9d0syT9F9AMc/edit?usp=sharing

Therefore:

Back-end actions (CC @JarrodBaker)

• [x] To "shut down" the generation of this annotation (cancerBiomarkers.scala).
• [x] Remove the cancerBiomarkers API endpoint

Front-end actions (CC @andrewhercules)

• [x] To remove the Cancer Biomarkers widget from the target page
• [ ] To accommodate new data source: widget + table

[andrewhercules]

@ireneisdoomed, I have updated #1645 and have asked the front-end team to remove the Cancer Biomarkers summary widget and detail view from the target profile page.

When evidence details are ready, including the relevant fields and datasource name (e.g. Cancer Genome Interpreter) please let me know and I can create design specifications for a summary widget and detail view for the evidence page. The front-end team will also make sure the filters on the association page are updated. And we can adjust the documentation and include the new source on the evidence page.

I'm also CCing @HelenaCornu as we will want to explain this in our release comms if the change is ready for 21.09 :-)

[ireneisdoomed]

Chemical Probes Updates

Until now, chemical probes were generated by parsing the spreadsheet that the data team maintained and curated manually. From 21.09 on, Chemical Probes and information about the ProbeMiner score will come from probes-drugs.org, the integration of this resource being described in #1536.

The proposed model with all the different endpoints can be seen here (current version is iter3): https://docs.google.com/spreadsheets/d/1AqC6aqKgyf_s-R1LculocodpjcHRMw6t-pUoVsvryxs/edit?usp=sharing

The latest version of the output dataset is uploaded to the path we were using for ProbeMiner data (gs://otar001-core/ProbeMiner/annotation). I propose renaming the parent directory to accommodate

Actions:

• [x] Update the ChemicalProbes API endpoints @JarrodBaker

[JarrodBaker]

@ireneisdoomed just confirming relating to cancer biomarkers and your comment on 9 August: you don't want any cancer biomarker information to be available via the API?

[andrewhercules]

Ticket closed as target part of the 21.09 release