late phase 2: ingest TRAPI KP `/meta_knowledge_graph` edge qualifier info, and use it in sub-querying

[colleenXu]

From bullet 4 of the changelog:

In MetaKnowledgeGraph component, added MetaEdge.qualifiers as list of new MetaQualifier types https://github.com/NCATSTranslator/ReasonerAPI/pull/387/files

Currently, we don't know what qualifier-constraints a TRAPI KP can handle; when a QEdge has qualifier-constraints (and isn't an "inferred" creative-mode query), we pass along those qualifier-constraints to the TRAPI KPs in our subqueries.

However, with TRAPI 1.4, tools can specify the qualifier-constraints they can handle for each TRAPI-MetaEdge in their /meta_knowledge_graph endpoint responses (edges section).

We want to use this info: when a QEdge has qualifier-constraints, we want BTE to only send sub-queries to TRAPI KPs that advertise that they can handle those qualifier-constraints. This should make our sub-querying behavior more efficient. We probably want to change how we create BTE-operations/MetaEdges from parsing the /meta_knowledge_graph responses of the TRAPI KPs we use.

• timing: phase 2. Ideally, we will have "real" /meta_knowledge_graph responses to work with (from the TRAPI KPs we use, their TRAPI 1.4 instances or we could ask them for mock /meta_knowledge_graph output?)
• priority: low (within phase 2).
  • We should still be able to query TRAPI KPs successfully, even if we lack this functionality
  • This ingest is a prereq for this phase 3 issue

Notes:

• keep in mind the difference between x-bte annotation with qualifiers vs TRAPI KP MetaEdges with qualifiers:
  • when we parse a x-bte operation, the qualifier info is the exact set of type_ids/values for the associations/data
  • but when we parse a TRAPI KP MetaEdge, there is ambiguity and multiple applicable_values elements are possible....so this isn't an exact-matching scenario.
    • ambiguity: associations might exist with no qualifiers or a set of type_id/value pairs. That set may have 2 pairs,...up to the total number of unique type_ids, because each type_id can only appear once in a qualifier-set.
    • For each type_id, its value could be one of the elements in its applicable_values set
• maybe relevant older notes?
  • some comments on the previous behavior we implemented
  • various issues that we probably never need to worry about given how we do TRAPI KP sub-querying

[colleenXu]

More discussion of the TRAPI MetaEdge / qualifier-set point (the "keep in mind" note above) is being done in the other issue. We'll add the clarifications / answers from the TRAPI team and Translator there.

[colleenXu]

Also, based on a quick check, none of the TRAPI KPs we ingest so far seem to specify qualifiers in their MetaEdges right now (automat kps, cohd, chp).

• COHD's dev instance seems to be on TRAPI 1.4 (we can access it through the registration we currently use, but they also registered a separate yaml for TRAPI 1.4). But I didn't see any "qualifiers" specified for the MetaEdges in its /meta_knowledge_graph response

---

And This is a "technical implementation" thought: but maybe the logic can be "exclusion":

• "if this hop has qualifier_constraints AND this TRAPI KP MetaEdge specifies qualifiers"...
  • check if the qualifier_constraints are covered by the MetaEdge qualifiers
    • so iterate through each set of type_id/value pairs....are all members of the set covered
    • If all sets ARE NOT covered, then don't do this sub-query. since the TRAPI KP is basically advertising that it doesn't support those qualifier_constraints.

[colleenXu]

Restating some info from above: this issue is probably on-hold because it (1) currently don't have TRAPI KP data to work with, (2) is affected by the "qualifier-set merging" discussion, and (3) seems optional (optimizing)

[tokebe]

Just a note: above linked PRs are currently on Dev.

[colleenXu]

Appears to be working.

Testing process

TRAPI query

``` { "message": { "query_graph": { "nodes": { "n0": { "ids":["PUBCHEM.COMPOUND:2662"], "categories":["biolink:SmallMolecule"] }, "n1": { "categories":["biolink:Gene"] } }, "edges": { "e1": { "subject": "n0", "object": "n1", "predicates": ["biolink:affects"], "qualifier_constraints": [ { "qualifier_set": [ { "qualifier_type_id": "biolink:object_direction_qualifier", "qualifier_value": "decreased" } ] } ] } } } } } ```

* when the query above (which has qualifier-constraints) is sent to Automat-pharos (ITRB Prod) `https://automat.transltr.io/pharos/1.4/query`, there are 15 edges returned. * when the query above is sent to BTE, BTE's console logs show that 1 sub-query is generated to Automat-pharos and 15 records are retrieved (as-expected) * Spot-checking showed that the data from the Automat-pharos response was preserved in BTE's edges. BTE's console logs: ``` bte:call-apis:query query success, transforming hits->records... +11ms bte:api-response-transform:index api name Automat-pharos(Trapi v1.4.0) +32ms bte:api-response-transform:index api tags: translator,automat,trapi,bte-trapi +0ms bte:call-apis:query Successful POST https://automat.transltr.io/pharos/1.4 (1 ID): SmallMolecule > affects > Gene (obtained 15 records, took 887ms) +0ms ```

Edge from Automat-Pharos response

``` "494976": { "subject": "PUBCHEM.COMPOUND:2662", "object": "NCBIGene:9536", "predicate": "biolink:affects", "sources": [ { "resource_id": "infores:pharos", "resource_role": "primary_knowledge_source", "upstream_resource_ids": null, "source_record_urls": null }, { "resource_id": "infores:automat-pharos", "resource_role": "aggregator_knowledge_source", "upstream_resource_ids": [ "infores:pharos" ], "source_record_urls": null } ], "qualifiers": [ { "qualifier_type_id": "biolink:object_aspect_qualifier", "qualifier_value": "activity" }, { "qualifier_type_id": "biolink:object_direction_qualifier", "qualifier_value": "decreased" }, { "qualifier_type_id": "biolink:qualified_predicate", "qualifier_value": "biolink:causes" } ], "attributes": [ { "attribute_type_id": "biolink:Attribute", "value": "IC50", "value_type_id": "EDAM:data_0006", "original_attribute_name": "affinity_parameter", "value_url": null, "attribute_source": null, "description": null, "attributes": null }, { "attribute_type_id": "biolink:publications", "value": [ "PMID:26653180", "PMID:27554445" ], "value_type_id": "EDAM:data_0006", "original_attribute_name": "publications", "value_url": null, "attribute_source": null, "description": null, "attributes": null }, { "attribute_type_id": "biolink:Attribute", "value": 6.37, "value_type_id": "EDAM:data_0006", "original_attribute_name": "affinity", "value_url": null, "attribute_source": null, "description": null, "attributes": null } ] } } ```

BTE preserves sources / attributes / qualifiers from Automat-pharos response

VS corresponding Edge from BTE's response

``` "46a372050469e5a98b9596619576c134": { "predicate": "biolink:affects", "subject": "PUBCHEM.COMPOUND:2662", "object": "NCBIGene:9536", "qualifiers": [ { "qualifier_type_id": "biolink:object_aspect_qualifier", "qualifier_value": "activity" }, { "qualifier_type_id": "biolink:object_direction_qualifier", "qualifier_value": "decreased" }, { "qualifier_type_id": "biolink:qualified_predicate", "qualifier_value": "biolink:causes" } ], "attributes": [ { "attribute_type_id": "biolink:Attribute", "value": "IC50", "value_type_id": "EDAM:data_0006", "original_attribute_name": "affinity_parameter", "value_url": null, "attribute_source": null, "description": null, "attributes": null }, { "attribute_type_id": "biolink:publications", "value": [ "PMID:26653180", "PMID:27554445" ], "value_type_id": "EDAM:data_0006", "original_attribute_name": "publications", "value_url": null, "attribute_source": null, "description": null, "attributes": null }, { "attribute_type_id": "biolink:Attribute", "value": 6.37, "value_type_id": "EDAM:data_0006", "original_attribute_name": "affinity", "value_url": null, "attribute_source": null, "description": null, "attributes": null } ], "sources": [ { "resource_id": "infores:pharos", "resource_role": "primary_knowledge_source", "upstream_resource_ids": null, "source_record_urls": null }, { "resource_id": "infores:automat-pharos", "resource_role": "aggregator_knowledge_source", "upstream_resource_ids": [ "infores:pharos" ], "source_record_urls": null }, { "resource_id": "infores:biothings-explorer", "resource_role": "aggregator_knowledge_source", "upstream_resource_ids": [ "infores:automat-pharos" ] } ] }, ```

This same testing process (same query) worked to review automat hetio behavior as well (4 edges in that KP's response -> 4 records in BTE's logs and 4 edges in BTE). Automat hetio's meta_knowledge_graph shows that there's a SmallMolecule - regulates - downregulated (qualifier) Gene MetaEdge. So BTE likely used that MetaEdge to generate the sub-query; this shows that qualifier-hierarchy expansion is working (since I queried for the parent qualifier decreased).

Notes:

• COHD and CHP meta_knowledge_graph endpoint responses show no qualifier-constraints, so not able to test.

Closing issue