x-bte annotation refactoring discussion

[colleenXu]

Going to open this issue as a place-holder for now. Look in comments for discussion on what is involved here

[colleenXu]

Idea from @tokebe:

• move query-handler config on edge attributes for hashing (unique edges), so somehow KPs can tell BTE how to process their edges / use some edge-attributes to avoid merging Edges
  • like a field allowing you to provide a list of attribute_type_ids?
  • only for Multiomics/Text-Mining (ones that provide edge-attributes) or for all?

Ideas from me:

• remove unneeded fields (useTemplating)
• redefine what counts as 1 written "unit of x-bte annotation" (operation? vocab to-be-defined):
  • uncouple what's written in the YAML vs what representation exists within BTE (MetaXEdge?)
    • "automatic expansion" annotation and flag. so 1 operation -> many in BTE potentially
    • design? what's written in the YAML should be human-writable, simple -> not look like code! which may be tricky with the nunjucks / jq work...
    • shouldn't be too dependent on a specific TRAPI/biolink-model version...
  • what counts as 1 unit?
    • 1 unique combo of subject-semantic-type / predicate / object-semantic-type
    • what about different ID-namespaces for input and output, and qualifiers?
      • how do we handle tweaks in the way you query (parameters / requestBody / requestBodyObject) and parse responses (response-mapping)
  • how to handle edge-attributes / qualifiers (especially mapping to biolink-model / TRAPI standards)

[colleenXu]

Another idea, although may be more "modifying / adding operations": using list_filter for BioThings APIs more, although that doesn't fully fix reverse-operation issues #316

[colleenXu]

From a convo Jackson and me had today:

Some main things to address?

• multiple-prefixes for inputs and outputs (aka the inputs + requestBody + parameters restructure, and outputs + response-mapping restructure)
• qualifier-sets (multiple sets per unit?? -> tied to requestBody / parameters?? list_filter/jmespath??)
• underlying sources stuff (multiple possible values, more than 1 value per record, list_filter / jmespath???)

Note:

• JQ-stuff only for basic stuff (mental model / more complex stuff should be BTE stuff) -> we don't want x-bte annotation stuff to look basically like code

[colleenXu]

@newgene This is the issue on x-bte refactoring. I heard during today's group meeting that there's some issues with MetaKG stuff and id-prefixes / operation-level source.

However, @tokebe and I have discussed the way x-bte annotations are written may be set up 1 way to be more writer-friendly, and be parsed into a different representation for BTE's MetaKG.....because there are diff requirements for both or design ideas.

I wonder if this is the case for your Translator/SmartAPI-Registry's MetaKG work....I suspect this work has a different set of requirements from BTE's internal representation AND the x-bte annotation writing...

[colleenXu]

One case of "multiple prefixes in output" is https://github.com/biothings/biothings_explorer/issues/585

[colleenXu]

Jackson @tokebe and I think this is the overarching topic for x-bte refactoring (after reviewing the notes above):

The issues

There seems to be 3 different requirement sets at play, that we want to tell apart and be aware of:

• "writer-friendly x-bte annotation":
  • easy to write/teach/maintain, can write manually (without using code or UI)
  • shouldn't be completely like code
  • has clear expectations for format / allowed values / what everything is used for
  • flexible, expressive
  • not dependent on specific TRAPI / biolink-model stuff that's still in-flux
• "internal BTE use": what BTE needs to keep track of all the info, construct sub-queries, edge management, etc. (vocab: BTE MetaEdge, MetaXEdge, bteEdge...)
  • x-bte annotation may be too "collapsed" from this POV, and BTE will need to expand 1 operation -> multiple internal representations
• "SmartAPI Registry MetaKG use": https://smart-api.info/portal/translator/metakg. What's needed for this tool / UI
  • x-bte annotation may be too verbose/specific from this POV, and this'll need to collapse multiple operations -> 1 MetaEdge for its purpose

Which leads to specific questions for group discussion, like:

• How does "1 x-bte operation / unit of annotation" relate to similar concepts (MetaEdges?) in BTE and SmartAPI Registry MetaKG?
  • and how does x-bte refactoring relate to and potentially change this?
• are BTE and SmartAPI Registry MetaKG using the same code? Does that make sense or should they use different code to process x-bte annotations?

---

And some ideas on how to "expand" an x-bte operation/ unit of annotation

Currently, 1 x-bte operation represents...

* 1 API endpoint being used * 1 unique combo of: * input semantic-type * input ID namespace * sub-query information * predicate * qualifier-set * source field value * output semantic-type * output ID namespace

Jackson @tokebe and I have discussed how to make it easier to write x-bte annotation - and one of our ideas is to have 1 x-bte operation (one unit of annotation?) expand to include more info:

• first-step proposal is #748
  • since there can be "combinatorial explosions" of current operations where the main difference comes from the input/output ID namespaces
• Other sources of "combinatorial explosions" are:
  • unique qualifier-sets
  • unique source field values
• note that all of these aren't as easy as "list out the possible values". There can be sub-query info, response-mapping info, post-processing info differences based on unique value/set...

my qualifier-set thinking

There are theoretically many operations that would mainly differ by qualifier-set (and how that affects sub-query info like post_filter/filter, jmespath, JQ). The guidance for [anatomical](https://github.com/biolink/biolink-model/blob/db44be0c49939229c28cbb71a715127941e0ce0b/biolink-model.yaml#L1515) / [species](https://github.com/biolink/biolink-model/blob/db44be0c49939229c28cbb71a715127941e0ce0b/biolink-model.yaml#L1532) / and [population](https://github.com/biolink/biolink-model/blob/db44be0c49939229c28cbb71a715127941e0ce0b/biolink-model.yaml#L1158) context qualifiers is currently unclear to me (are they edge-attributes or part of the qualifier-set?). If they turn out to be part of the qualifier-set and we want to suppor them, this has combinatorial explosion problems because the context qualifiers in our KPs have a lot of possible values. * anatomical context: * multiomics apis (drug response): Guangrong has previously told me that some operations are affected, and include 10-20s of possible tissue/anatomical-context values * also in pending apis: ebi gene2pheno * species context: affects lots of apis * core biothings: MyChem chembl.drug_mechanism and drugcentral.bioactivity info, MyGene panther, a little MyDisease disgenet) * pending biothings: bindingdb, mgi gene 2pheno * external: ctd, biolink/monarch * population context: * multiomics apis based on clinical data: ehr risk, wellness (clinical trials too?)

My source field thinking

There are theoretically some operations that would mainly differ by source (and how that affects sub-query info like post_filter/filter, jmespath, JQ...). It would be nice if we could set the source info to field values that are post-processed by BTE... I'm not sure of the scope of this issue though: * core biothings apis: mygene, mydisease disgenet * external apis: biolink/monarch Also maybe complicated because some api hits will have multiple source values / fields?