Ensure reactome go-cams validate against main GO shex schema

[goodb]

Revisit and rework conversion rules or shex schema to bring them into alignment. Close when everything passes. Re-open individual tickets for future failures.

• [x] Implement re-usable software framework for testing models that is integrated with this repository's travis system
• [ ] Complete a clean validation run using this framework

[goodb]

@deustp01 I am so sorry, I mistakenly generated that file with plant reactome biopax as the input. Glad you figured out a better way, and really sorry if I wasted any of your time. In case its still useful, here is the human untyped entity file. untyped_physical.txt

I'm not sure if chebi:role is a good default for untyped entities. We do already handle some cases of chebi entities that are roles and not entities - e.g. anti-inflammatory drugs https://www.ebi.ac.uk/chebi/chebiOntology.do?chebiId=CHEBI:35475 . If you have to group things like "cold temperature" and "long day photoperiod" together I think it might be better to just leave them untyped. For shex purposes, if something typed as a chebi:role is in a position where we are expecting e.g. protein or information biomacromolecule, it won't validate anyway.

[deustp01]

No waste. The plant instances need to be fixed as well in the long run, and even in the short term it could be a neat demonstration of the generality of the Pat2GO scheme that it supports a reasonably clean, low-loss rice export as well as human.

I'll undo the ChEBI:role typings. Only a few plant instances are affected; all human and species-agnostic ones got valid chemical IDs, I think.

[deustp01]

@goodb The new untyped_physical.txt file includes set entities as well as individual entities. Even when all members of a set fall in the same chemical class, e.g., they are all proteins or small molecules, it is safer (and maybe logically required) to assign a cross-ref to the set itself to tell shex what kind of chemical it contains? I guess complexes escape this requirement because all our complexes have protein components so it is reasonable to type the complex as "protein" even if it also sometimes contains non-protein subunits? Or "complex" is its own type in shex?

[ukemi]

I thought GO protein-containing complex was also valid. Seems like this would fit better.

[ukemi]

I also don't think it is a bad thing to try this with the plant Reactome. Although I think it is out of scope for this project, it is a next logical step at extending this effort to other pathway representations. I would think that plant Reactome is a baby step that should prett much work out of the box with minimal tweaking. Let's create a roadmap in NYC for what steps we need to take once this first step is complete.

[deustp01]

I thought GO protein-containing complex was also valid. Seems like this would fit better.

Do I need to add this term as an xref anyway to Reactome complex instances? Can't my reasoning above be used to tell shex that when it sees a Reactome complex and needs a classification, it should use GO:0032991 "protein-containing complex", no manual case-by-case manual intervention needed?

Also, is it OK to create xrefs for some instances with ChEBI terms and for others with GO_cell_component terms? That won't break any consistency checks at GO-CAM? Except for the manual work involved, it's no problem on our end - these xrefs are just used to make links out to external resources from the web page that shows the details of the instance so a user browsing from instance to instance might wonder at the mixture of kinds of links but nothing should break.

[deustp01]

I also don't think it is a bad thing to try this with the plant Reactome. Although I think it is out of scope for this project, it is a next logical step at extending this effort to other pathway representations. I would think that plant Reactome is a baby step that should prett much work out of the box with minimal tweaking. Let's create a roadmap in NYC for what steps we need to take once this first step is complete.

Clarification - I thought Ben has already done a Path2GO conversion for the rice annotations, and it came out mostly OK. If (and only if) that's true, then it's worth a sentence or two in the paper and in a grant proposal to point to it as evidence that the conversion process can be generalized beyond human. In the same paragraph with a sentence or two pointing out that as the conversion relies on BioPAX it can be generalized beyongh Reactome format to anything that can be converted to BioPAX.

[ukemi]

You are correct. I think at the Berkeley meeting @goodb did the initial conversion and the models were generated. However, I don't think they were validated. I think it would be worth mentioning, but I am trying to avoid scope-creep so that we can have a defined end-point to this first phase of the project.

[deustp01]

Agreed - no scope-creep; only what can be used as-is now (if anything).

[goodb]

@deustp01 there is already logic in the converter that will assign types to sets based on their components, so no need to do that manually from the conversion perspective.

Regarding plant reactome, we got as far as successfully running the converter, without modification, on the plant reactome biopax export and generating go-cams for all the pathways. Shex and OWL validation could be done trivially if we want, though should await any in-progress changes to address known problems. Manual inspection of one pathway looked good. Probably enough already for a sentence or two. If we want to flesh the generality aspect out further, we could spend a bit of time running things on pathway commons again.

Here is ABA synthesis in rice from the conversion at the meeting.

[goodb]

I thought GO protein-containing complex was also valid. Seems like this would fit better.

Do I need to add this term as an xref anyway to Reactome complex instances? Can't my reasoning above be used to tell shex that when it sees a Reactome complex and needs a classification, it should use GO:0032991 "protein-containing complex", no manual case-by-case manual intervention needed?

Correct, no manual case-by-case is needed here. Complexes are already detected and assigned to the protein-containing complex class. I would suggest that any xrefs used for link outs here point to PRO or complex portal entities if they exist. But I guess that is for the future.

Also, is it OK to create xrefs for some instances with ChEBI terms and for others with GO_cell_component terms? That won't break any consistency checks at GO-CAM? Except for the manual work involved, it's no problem on our end - these xrefs are just used to make links out to external resources from the web page that shows the details of the instance so a user browsing from instance to instance might wonder at the mixture of kinds of links but nothing should break.

Yes, the OWL and shex logic should handle GO CC and CHEBI fine. Use as you see fit. The only tricky part here are the roles. These can be used (e.g. the drug classes I mentioned earlier) but may require some intervention - I'll have to look to see what happens if they are added.

[deustp01]

roles - it's safe to leave this for the future

[deustp01]

Here is Ben's spreadsheet from November 14.

I've checked all items.

For genomeEncodedEntities (GEEs, informational macromolecules that cannot be associated with a UniProt or Ensembl ID) and otherEntities (just that - things known to be discrete physical entities but the information to map them to single reference chemicals or macromolecules is missing) I've either confirmed that a high-level ChEBI term (e.g., ChEBI:16991 "deoxyribonucleic acid" or even ChEBI:24431 "chemical entity) has been added, or for instances not found in previous searches of gk_central, I've added them.

Previously, I overlooked our polymer class. We use this mostly to annotate well-defined protein multimers like microtubules and collagen structures, but we also have polyguanosine. The presence of covalent bonds attaching the monomers to one another is optional. Almost all, I think can be tagged with ChEBI:36080 "protein" as a cross-reference. (For version 2, it might be interesting to ask the script to try to retrieve specific UniProt or ChEBI identifiers from the annotations of the polymer components and use them to classify the polymer more precisely, but that is definitely a refinement for later.)

I also previously overlooked a number of non-human GEEs, mostly viral and bacterial entities that serve to trigger processes of normal innate immunity. These should all have cross references now.

Per Ben's comment above, I have not added cross references to set instances but have indicated the appropriate high-level ChEBI term in the spreadsheet in case that's useful for checking validation results or for training the validator. (In some cases, e.g., a set of complexes of complexes, it's necessary to drill down several steps to find an identifier that establishes the chemical nature of the identifier.) untyped_physical_Hs.xlsx

[goodb]

Looks good @deustp01 . I think right now, we mainly need a response from Guanming about why these higher-level xrefs are not appearing in the curator-site biopax exports.

Did you look through the reactions with multiple enabling entities ? That was the other main class of problems with the shex.

[deustp01]

Did you look through the reactions with multiple enabling entities ? That was the other main class of problems with the shex.

Oops, no. Point me to the list / comment and I will get to work on that.

[ukemi]

Based on the GPAD outputs, we may want to revisit any type of cardinality rules we have here. We will need @deustp01's expertise on how various classes of enzyme complexes work.

[goodb]

@deustp01 here is what I had found when last I looked.

id title R-HSA-5696397 Gap-filling DNA repair synthesis and ligation in GG-NER R-HSA-5685939 HDR through MMEJ (alt-NHEJ) R-HSA-912446 Meiotic recombination R-HSA-6782135 Dual incision in TC-NER R-HSA-6783310 Fanconi Anemia Pathway R-HSA-6782210 Gap-filling DNA repair synthesis and ligation in TC-NER R-HSA-1445148 Translocation of SLC2A4 (GLUT4) to the plasma membrane R-HSA-5655862 Translesion synthesis by POLK R-HSA-5693565 Recruitment and ATM-mediated phosphorylation of repair and signaling proteins at DNA double strand breaks R-HSA-5693607 Processing of DNA double-strand break ends R-HSA-212300 PRC2 methylates histones and DNA R-HSA-427413 NoRC negatively regulates rRNA expression R-HSA-6807505 RNA polymerase II transcribes snRNA genes R-HSA-1660499 Synthesis of PIPs at the plasma membrane R-HSA-1660516 Synthesis of PIPs at the early endosome membrane R-HSA-1660514 Synthesis of PIPs at the Golgi membrane R-HSA-1660517 Synthesis of PIPs at the late endosome membrane R-HSA-1362409 Mitochondrial iron-sulfur cluster biogenesis R-HSA-1227986 Signaling by ERBB2 R-HSA-156842 Eukaryotic Translation Elongation R-HSA-9033241 Peroxisomal protein import R-HSA-6791226 Major pathway of rRNA processing in the nucleolus and cytosol R-HSA-72203 Processing of Capped Intron-Containing Pre-mRNA R-HSA-9027283 Erythropoietin activates STAT5 R-HSA-2559580 Oxidative Stress Induced Senescence R-HSA-5213460 RIPK1-mediated regulated necrosis R-HSA-416572 Sema4D induced cell migration and growth-cone collapse R-HSA-212676 Dopamine Neurotransmitter Release Cycle R-HSA-210500 Glutamate Neurotransmitter Release Cycle R-HSA-181429 Serotonin Neurotransmitter Release Cycle R-HSA-264642 Acetylcholine Neurotransmitter Release Cycle R-HSA-8866652 Synthesis of active ubiquitin: roles of E1 and E2 enzymes R-HSA-8866654 E3 ubiquitin ligases ubiquitinate target proteins R-HSA-2682334 EPH-Ephrin signaling R-HSA-2871796 FCERI mediated MAPK activation R-HSA-450302 activated TAK1 mediates p38 MAPK activation R-HSA-450341 Activation of the AP-1 family of transcription factors R-HSA-450321 JNK (c-Jun kinases) phosphorylation and activation mediated by activated human TAK1 R-HSA-9013973 TICAM1-dependent activation of IRF3/IRF7 R-HSA-168927 TICAM1, RIP1-mediated IKK complex recruitment R-HSA-937041 IKK complex recruitment mediated by RIP1 R-HSA-936964 Activation of IRF3/IRF7 mediated by TBK1/IKK epsilon R-HSA-983695 Antigen activates B Cell Receptor (BCR) leading to generation of second messengers R-HSA-1169091 Activation of NF-kappaB in B cells R-HSA-1679131 Trafficking and processing of endosomal TLR R-HSA-202433 Generation of second messenger molecules R-HSA-909733 Interferon alpha/beta signaling

[deustp01]

Thanks - that's what I needed.

[deustp01]

I've done 21 pathways on Ben's list. There appear to be only a few kinds of problems, which recur regularly as itemized in the attached Word document (with color coding that gets lost in a copy-paste to this comment window). From what I know of our curation styles / variability this sample should be enough to reveal most of the ways that we've found to break parsing of a Reactome catalystActivity attribute into GO-CAM so I'd like to turn to something else in preparation for the meeting. @goodb But let me know if it's important to get through the entire list now. events_with_mutiple_enablers.docx

[goodb]

@deustp01 I don't think you need to go through the whole list right now. I had hoped we might make it to a clean release before the meeting, but that doesn't seem feasible at this point. For now, I think its more important to work on defining the general patterns where things don't quite line up as it stands than to fix every instance.

I was a little unclear about two of your error categories though. Right now, the shex rules allow for the possibility of a Complex in the enabled_by slot for a reaction. So the grey (catalyst is a homomultimer) and yellow (catalyst is a complex) tags shouldn't indicate a shex-specific problem. They are probably most relevant for the GPAD generation, right??

[deustp01]

They are probably most relevant for the GPAD generation, right??

I don't know. I was cataloguing all the places where the active unit of the physical entity that enables catalysis molecular function is anything other than a UniProt protein and, indeed, going through the list there are no pathways all of whose issues are gray or yellow - they all have some green or purple as well, so it looks like our focus is only the green (sets as active units) and purple (more than one catalyst or active unit per reaction) ones. That makes sense, and points the gray and yellow instances to "Systematic Review of Pathway Conversion" #40.

[goodb]

Okay, on the same page. I was thinking about GPAD generation as these kinds of structures currently would not yield any gpad-style annotations to the proteins in the complexes and perhaps they should.

[ukemi]

I think you are correct, but this is happening in other scenarios as well. We should be sure to touch on this in the complexes discussion.

[goodb]

Dec. 18 shex/owl validation test Using new Reactome model collection, released late november. Using current converter - noting the return of Transport processes. Using pending shex file here: https://github.com/geneontology/go-shapes/pull/172 1806 models tested 197 have a shex validation error 0 have an OWL error

Many errors are caused by regulates relationships linking MFs to BPs

I think most other errors come from untyped entities including: inputs: R-ALL-9634466 D5v8 (herceptin) R-ALL-9665971 pertuzamab

enablers: R-HSA-175986 Thiol S-methyltransferase R-HSA-9661701 UBGNO R-CPE-9661737 UBGNR R-CPE-9661715 BILR R-HSA-9663470 BGET R-ALL-72617 eIF1

Looking at the untyped entities coming from the Reactome November 2019 build, there are quite a few and it seems that at least some that Peter has fixed did not make it through into this release. Not sure if that is a timing issue or a technical one. example: R-ALL-157771 ought to have the type 'protein' but that didn't come through in the latest release. Here is the current untyped list.

untyped_physical.txt

[ukemi]

The first two are monoclonal antibody cancer therapeutics. Are they inputs in normal patheways @deustp01? They would be proteins.

The others all look liked they can be typed too.

[ukemi]

Look on the bright side------no logical errors.

[ukemi]

I see now, the 2 mabs are annotated as inhibiting the receptor functions in the signaling pathways. When I look in the pathway browser, they appear to be in a nonphysiological patway, but if I just search on them it looks like they are part of the pathway.

[deustp01]

Hmm. The theory is that drugs are used to treat diseases, so drug instances (small molecule or protein or engineered protein) should not be appearing in normal events, and the Path2GO process should not be attempting to process any disease events. And even if a "normal" event involves a drug, it's not truly "normal" in the sense that all of normal reaction space exists and operates without any drugs as participants.

I will look some more at how we've annotated these to see if there's a pattern that would enable Path2GO to recognize and exclude these pseudo-normal reactions.

[deustp01]

I think most other errors come from untyped entities including: inputs: R-ALL-9634466 D5v8 (herceptin) R-ALL-9665971 pertuzamab

@goodb These two instances point to a Reactome data structure oddity. It is permissible to include a "disease" reaction in a "normal" pathway. We do this to annotate actions of drugs that interact with normal proteins performing their normal functions, e.g., small molecules that inhibit some feature of blood clotting. You already use our disease attribute at the pathway level to identify pathways with non-null slots and not process them. An additional step (easy to say) would be to apply that test again to identify and remove "disease" reactions within normal pathways. A test that probably overlaps with the first one is to discard any reaction whose participating entities include ones that belong to the chemicalDrug or proteinDrug classes. (In principle, the two tests should flag the identical reactions - in reality??)

I expect that removal of all these disease reactions would leave a normal reaction set that still has the full connectivity of the pathway - yell if that's not true! Even if we've avoided that trap, this way of annotating drug functions may not be maintainable so anything that comes out of the Path2GO project that suggests additional problems it causes would be good to know to help us figure out a better annotation strategy.

Here's an example, for R-HSA- 9665971 pertuzamab

[ukemi]

Maybe easier said than done. We can't use the ChEBI role drug or we will remove all reactions that use ATP. :) Do you know of another way to identify drugs more rigorously?

[deustp01]

I'm suggesting we use the Reactoime annotation data themselves: our identification of a reaction as a disease event even though it is part_of a normal pathway, and / or the presence of a physical entity of the chemicalDrug, proteinDrug, or RNADrug Reactome classes as a participant in the reaction.

Agreed, looking up Reactome entities in ChEBI to identify their ChEBI-assigned roles would not work.

[ukemi]

Sweet. The power of annotation.

[goodb]

@deustp01 Actually I don't use the Reactome disease annotation slot for pathway filtering because it does not come through in the BioPAX. I have hackily filtered out the disease pathways simply by looking for occurrences of 'disease' in the pathway name. This annotation also does not come through at the Reaction level. So.. if we wanted to make use of it in the converter, we would need to talk to Guanming about encoding it somehow. It could perhaps be added as an additional xref linking to some appropriate ontology class or (less good) another parsable comment.

If we do attempt to filter at the reaction level, it looks like this will be disruptive to some pathways. As an example, I'm looking at reaction R-HSA-1963589 ERBB2 forms heterodimers with ligand-activated ERBB receptors. It has a reference to pertuzamab. If we took that reaction out of the Signaling by ERBB2 pathway, we would lose connectivity from its upstream 'ERBB3 binds neuregulins ' to its downstream 'SRC family kinases phosphorylate ERBB2'

[ukemi]

Rats. I agree we can't remove that reaction. Would it work to remove anything with 'disease' or 'drug'? Still hacky but might work.

[goodb]

As far as the code for the converter is concerned, I don't see why it should not be able to handle pathways involving drugs and diseases. We might or might not want to avoid including them in the GO knowledge base but that is a separate issue.

I think we should focus first on getting these entities typed. I think the IUPHAR annotations ought to make that possible one way or another.

[deustp01]

As an example, I'm looking at reaction R-HSA-1963589 ERBB2 forms heterodimers with ligand-activated ERBB receptors. It has a reference to pertuzamab. If we took that reaction out of the Signaling by ERBB2 pathway, we would lose connectivity from its upstream 'ERBB3 binds neuregulins ' to its downstream 'SRC family kinases phosphorylate ERBB2'

True, but that regulatory interaction is specifically due to the drug interaction - no drug, the normal target reaction proceeds unregulated, so I think it's OK to lose it as a piece of "normal" biology.

[ukemi]

But even though the reaction is negatively regulated by the drug, the R-HSA-1963589 reaction itself represents the normal receptor-ligand interaction doesn't it?

[deustp01]

I think we should focus first on getting these entities typed. I think the IUPHAR annotations ought to make that possible one way or another.

All our chemicalDrug instances have IUPHAR instances as referenceEntity attributes, as well as ChEBI instances as cross-refs, and presence of both attributes is an editorial requirement. All eight of our proteinDug instances (all monoclonal antibodies) have IUPHAR referenceEntity attributes. Are goal (achieved so far) is to have an IUPHAR reference for every drug we release. Is that information accessible in the BioPAX export, and usable here to identify reactions that are unwanted for now?

[ukemi]

I think that the problem is that since it references the pertuzamab as a negative regulator, a simple filtering of any reaction mentioning pertuzamab won't work.

[ukemi]

I knew this was going to be harder than it looked. :)

[ukemi]

If the IUPHAR info is there, that should do the trick. A Reactome curator would not have assigned and IUPHAR value unless the entity was acting as a drug, right?

[deustp01]

But even though the reaction is negatively regulated by the drug, the R-HSA-1963589 reaction itself represents the normal receptor-ligand interaction doesn't it?

No. Normally, ERBB2:ERBIN:HSP90:CDC37 interacts with Ligand-Activated EGFR/ERBB3/ERBB4 to form an array of outputs. Alternatively ERBB2:ERBIN:HSP90:CDC37 can bind any of an array of drugs to form complexes that have no further function, sequestering ERBB2:ERBIN:HSP90:CDC37. In the specific case where ERBB2:ERBIN:HSP90:CDC37 binds pertuzumab drug, the resulting complex inhibits the normal binding reaction, so it not only sequesters a required input to the normal reaction but also prevents other unsequestered copies of the input complex from doing their normal binding.

[deustp01]

If the IUPHAR info is there, that should do the trick. A Reactome curator would not have assigned and IUPHAR value unless the entity was acting as a drug, right?

Yes. It's not perfect because it's an editorial standard not a hard-wired feature of the data model but we are working to keep this usage clean exactly as you say.

[ukemi]

No. Normally, ERBB2:ERBIN:HSP90:CDC37 interacts with Ligand-Activated EGFR/ERBB3/ERBB4 to form an array of outputs. Alternatively ERBB2:ERBIN:HSP90:CDC37 can bind any of an array of drugs to form complexes that have no further function, sequestering ERBB2:ERBIN:HSP90:CDC37. In the specific case where ERBB2:ERBIN:HSP90:CDC37 binds pertuzumab drug, the resulting complex inhibits the normal binding reaction, so it not only sequesters a required input to the normal reaction but also prevents other unsequestered copies of the input complex from doing their normal binding.

Got it!

[deustp01]

enablers: R-HSA-175986 Thiol S-methyltransferase R-HSA-9661701 UBGNO R-CPE-9661737 UBGNR R-CPE-9661715 BILR R-HSA-9663470 BGET R-ALL-72617 eIF1

Looking at the untyped entities coming from the Reactome November 2019 build, there are quite a few and it seems that at least some that Peter has fixed did not make it through into this release.

The first item in the list of enablers has a ChEBI xref in Reactome; the rest do not, probably because I overlooked them in the re-edit last month to add high-level ChEBI terms as xrefs to Reactome physicalEntities that are incompletely specified, e.g., we know it's a protein but not enough about its sequence to map it to a UniProt entry.

All the other items do have ChEBI xrefs, e.g., to ChEBI:36080 "protein". That's also true for the first 20 non-set items listed in untyped_physical.txt, so these cross-refs are not getting through to this release. (I only checked these 20 but they look representative.)

[goodb]

@deustp01 I just revisited the thread with Guanming and asked him how we might access this information. I can see it there in the curator site, but not in the public one or in any of the BioPAX. I suspect that he will need to make a specific release for us.

Let me know what you two think about including/excluding reactions containing drugs. If we want to go that route, I will set up the IUPHAR-based filter.

[deustp01]

@ukemi For Path2GO version 1, I think we exclude reactions that have drugs as participants, but bringing them in at least on a pilot scale, maybe as a first step towards bringing in disease processes / abnormal phenotypes, is a high priority for version 2 (and a discussion point for version 1, that the Path2GO tool can easily accommodate this extension).

[ukemi]

I agree completely with @deustp01. Including the reactions with drugs goes against general GO policy and is outside of scope. In the long term, this might be a very exciting extension of the project that goes beyond GO but moves further along the path towards complete integration of multiple resources.

[goodb]

Update. (Ping @vanaukenk regarding shex issues below).

I ran the conversion using the current rule set (which includes the use of transport and protein complex disassembly as types for certain reactions), against a January 2020 version of Reactome generated from the curator (pre-release) database.

1806 models processed . Zero OWL errors. 517 with some shex problem.

Here is a visualization of the shex conflicts:

Key Problems:

1. In the conversion, complexes are used as inputs, outputs, and objects of transport for Biological Process nodes. (Either transport or dissociation processes). This conflicts with the schema for Biological Process which only allows chemical entities or anatomical entities as inputs/outputs and only allows Information Biomacromolecules (genes, proteins) to be transported.
2. In the conversion, Molecular Function nodes are assigned causal (regulates, provides input for) relations to Biological Process nodes. This violates the MF schema which dictates that these relations only connect to other MF nodes.
3. We still have a variety of untyped physical entities - many of which seem to be Sets of mixed type.

I am going to work on 3. between now and the meeting tomorrow (as we will likely mostly talk about physical entities). @vanaukenk can you confirm whether or not the shex schema might change with regards to 1.? I believe our resolution on the shex call last week was that its going to stay as it is for the MF-BP relations (2. above) - assuming that is still the case, @ukemi could you check the model proposed here: https://github.com/geneontology/pathways2GO/issues/75#issuecomment-572252975 ?

[vanaukenk]

@goodb Wrt 1, it seems reasonable to me to allow complexes to be inputs, outputs, and objects of transport for Biological Process nodes. For 2, that is my understanding as well. We should discuss the proposed new model in #75 on the 2020-01-15 call.