overview and management of TRAPI 1.4 features

[colleenXu]

[WILL BE UPDATED AS WE DISCUSS; last updated 2022-05-22]

This is an overview of what the TRAPI 1.4 topics are, what their github issues are, and their statuses.

General topics

Consensus reached within our team:

• right now, our dev is in-flux (partially TRAPI-1.4) and may break (as other tools also move to TRAPI 1.4)
  • would be nice to inform other teams as our dev changes (because they use us). But perfect integration between our teams is not a realistic goal on dev (ever)
  • don't have to make TRAPI-1.4 yamls for our tools until we're asked / further along in our dev instance's implementation of TRAPI 1.4
• Multiomics / Text-Mining: how to keep TRAPI 1.3 + TRAPI 1.4 compliance
  • we should be able to do this, see the issue that covers their provenance situations for details

1. Async endpoints - done, but see note

• [x] covered by this issue (phase 1)

Note: not advertised yet (since we haven't updated SmartAPI yamls for our tools -> which is in section 5 Misc)

2. qualifiers in /meta_knowledge_graph - done

• [x] advertising x-bte operation qualifier info (phase 1)

Less important:

• [x] ingest and use TRAPI KP /meta_knowledge_graph qualifier info (phase 2)
• [x] ensuring that TRAPI KP qualifier info is in our general-ARA /meta_knowledge_graph response (phase 3)
• [x] qualifier-hierarchy handling (phase 3)

Note: Assuming that "qualifier-set" merging is required. Based on TRAPI/Translator group (Slack poll) on whether to do "qualifier-set merging" (our questions and references)

3. provenance refactor - done, but see notes

• [x] for x-bte operations (phase 1)
• [x] for some multiomics KPs, text-mining KP, TRAPI KPs (phase 1)

Notes:

• Multiomics and Text-Mining KPs are responsible for updating their BioThings APIs and creating separate SmartAPI yamls
• We have a different API_LIST + SmartAPI overrides for TRAPI 1.4 KPs and Multiomics/Text-Mining KPs TRAPI 1.4 support

4. aux-graph / result.analyses refactor - ongoing, on node-expansion/subclassing

Decision was made to implement TRAPI 1.4 "without" node-expansion/subclassing first, and then work on this part last.

First round, these issues WITHOUT node-expansion/subclassing work:

• [x] basic querying (phase 1)
• [x] processing TRAPI-1.4 KP responses (phase 1)
• [x] creative-mode (phase 2)

Second round, these issues with node-expansion/subclassing work:

• [x] basic querying (phase 1)
• [x] processing TRAPI-1.4 KP responses (phase 1)
• [x] creative-mode (phase 2)

5. Misc - ongoing, only the SmartAPI yaml part is critical

May be important to have the yaml to advertise the TRAPI 1.4 instances, and then the rest aren't as critical?

• [x] updating SmartAPI yamls (phase 3, as needed)

Less important:

• [x] trapi / biolink version specified in each response (phase 3, optional?)

Related but much less important

• association property for MetaEdges (phase 3, optional)
• readme/webpage on scoring method and creative-mode (phase 3 / later)

[colleenXu]

From today's group meeting:

What's checked off is done, don't need to think about it unless bugs arise JC is working on provenance (3), some aux-graph/results stuff that isn't subclassing (4), and directing Rohan on qualifiers (2)

Qualifiers: poll isn't complete, but we can move forward with "MetaEdge has union of qualifier sets" -> means we can work on the first two issues in that section

CX: I think we want to support sources ingest from Multiomics/Text-Mining ASAP (specific part of provenance work). Since their APIs will provide this info soon (BioThings API parser updates discussed in Translator #bte channel)

[colleenXu]

And a note regarding PFOCR "augmentation" work: https://github.com/biothings/biothings_explorer/issues/538 and https://github.com/biothings/biothings_explorer/issues/420...

Unclear how results-refactoring (4) affects this work...

• hopefully no effect since it uses node-bindings?
• But we'll see...(me: affected by aux-graphs???)

[colleenXu]

@tokebe Here's some possible test-queries for node-expansion. Let me know if you need more.

[UPDATED 2023-05-22 1:00PM Pacific]

2-hop explain, 1 QNode ID is non-primary

My notes are from using the main branch code (TRAPI 1.4 without subclassing) * ran in 2 min 5 sec * Noonan syndrome is using a non-primary ID (DOID). It expands to 14 IDs * Headache is using the primary ID. It expands to 10 IDs * got 37 results. * some have Noonan syndrome vs some have a descendant of Noonan syndrome * some have headache vs some have a descendant of headache The response I have: [2-hop-noonan-headache_2.txt](https://github.com/biothings/biothings_explorer/files/11536435/2-hop-noonan-headache_2.txt) ``` { "message": { "query_graph": { "edges": { "e00": { "subject": "n0", "object": "n1" }, "e01": { "subject": "n2", "object": "n1" } }, "nodes": { "n0": { "ids": ["DOID:3490"], "categories": ["biolink:Disease"], "name": "noonan" }, "n1": { "categories": ["biolink:PhenotypicFeature"] }, "n2": { "ids": ["HP:0002315"], "categories": ["biolink:PhenotypicFeature"], "name": "headache" } } } } } ```

creative-mode disease -> chem

New example: start with the non-primary ID for COPD: `DOID:3083` (primaryID is MONDO:0005002). * Results will include the descendant diseases like pulmonary emphysema (DOID:9675 / MONDO:0004849) and interstitial emphysema (DOID:10030 / MONDO:0000923). * Runs in ~ 56 s, only uses the first template Response in prod / TRAPI 1.3: [creative-copd.txt](https://github.com/biothings/biothings_explorer/files/11537583/creative-copd.txt) ``` { "message": { "query_graph": { "nodes": { "n0": { "categories":["biolink:ChemicalEntity"] }, "n1": { "ids":["DOID:3083"], "categories":["biolink:DiseaseOrPhenotypicFeature"], "name": "copd" } }, "edges": { "e0": { "subject": "n0", "object": "n1", "predicates": ["biolink:treats"], "knowledge_type": "inferred" } } } } } ```

My first example notes are here. I don't think it's working anymore since all results from the prod instance of BTE (TRAPI 1.3) and local instance main branches (TRAPI 1.4) are connected only to the originalID.

My notes are from using the main branch code * ran in 1 min 58 sec * started with the primary ID -> expands to 3 IDs (the other equivalent non-primary IDs like [DOID:3138](https://disease-ontology.org/?id=DOID:3138) and UMLS:C0000889 wouldn't expand...) * got 500 results from 2 templates, and all 3 IDs (so original and expanded) show up in the results * original acanthosis nigricans (MONDO:0007035) is in the 4th result Physostigmine (PUBCHEM.COMPOUND:5983) * expanded ID MONDO:0008696 (acanthosis nigricans-insulin resistance-muscle cramps-acral enlargement syndrome) is in the 1st result Urofollitropin (PUBCHEM.COMPOUND:62819) * other expanded ID MONDO:0043003 (familial acanthosis nigricans) is in the 40th result (which IS AN EDGE CASE that might be tricky) tretinoin Topical Cream (UMLS:C1252062). See screenshot below  The response I have: [creative-acanthosis.txt](https://github.com/biothings/biothings_explorer/files/11513882/creative-acanthosis.txt) ``` { "message": { "query_graph": { "nodes": { "n0": { "categories":["biolink:ChemicalEntity"] }, "n1": { "ids":["MONDO:0007035"], "categories":["biolink:DiseaseOrPhenotypicFeature"], "name": "acanthosis" } }, "edges": { "e0": { "subject": "n0", "object": "n1", "predicates": ["biolink:treats"], "knowledge_type": "inferred" } } } } } ```

[colleenXu]

@tokebe also just in case, here's the infores IDs for each ontology in node-expansion right now

They aren't always the same as the ID-prefix :P

GO -> infores:go DOID -> infores:disease-ontology MONDO -> infores:mondo CHEBI -> infores:chebi HP -> infores:hpo UMLS -> infores:umls

[tokebe]

With subclassing done, we're at a point where this issue is basically remaining for tracking any last-minute bugfixing, and waiting to close when TRAPI 1.4 is deployed to Prod.

[colleenXu]

In today's meeting, the main way we'll know that we've implemented everything for TRAPI 1.4 is if we pass TRAPI 1.4 validation in the ARAX UI (https://github.com/biothings/biothings_explorer/issues/587#issuecomment-1611893691).

However, I'd also like to check:

• has there been any changes to the TRAPI 1.4 spec since April 13th (last time I updated this issue)? If so, do we need any feature changes or just updates to the SmartAPI yamls?
• are there any upcoming changes to the TRAPI 1.4 spec? Is a full release of TRAPI 1.4.0 coming (not beta)?

[colleenXu]

Regarding what else is needed to deploy TRAPI 1.4 to Prod (all instances on TRAPI 1.4)...

1. Haven't discussed yet: update registered yamls for BTE / Service Provider and remove the temp registrations for TRAPI 1.4 instances? Unclear if we'll need to update code (SmartAPI exclusions?) @tokebe
2. Remove overrides / update registered SmartAPI yamls for text-mining + multiomics KPs
   1. we agreed that these overrides are temporary and we want to remove them in prep to getting all instances on TRAPI 1.4
   2. @tokebe will handle removing overrides and deploying the changes
   3. I'll handle the updating of SmartAPI yamls / refreshing registrations. This involves coordination with Jackson + giving heads-up to those teams. I'll likely take control, since this is a special case and timing is important
   4. keep the files used for overrides! Then BTE's Test instance will remain working throughout this process
   5. Once registered yamls are updated / registrations refreshed, BTE's Prod instance responses will be a bit wonky (sources showing up in edge-attributes) until Prod instance is updated

[colleenXu]

TRAPI spec 1.4.2 reviewed and updates made to SmartAPI yamls https://github.com/biothings/biothings_explorer/issues/594#issuecomment-1629687393, no other changes needed. Related to my earlier comment https://github.com/biothings/biothings_explorer/issues/613#issuecomment-1611996930

[tokebe]

Closing this -- the last few stragglers are more separate from this major push, now.