search

NCATSTranslator / ReasonerAPI

NCATS Biomedical Translator Reasoners Standard API
34 stars 28 forks source link

Result format (including ranking/scoring) #220

Closed colleenXu closed 3 years ago

colleenXu commented 3 years ago

Exploring Agent has noticed some inconsistencies and potential issues with the Result schema.

edeutsch commented 3 years ago

I think there's an additional optional property some ARAs are providing to share all the "paths" going to a result node in the knowledge graph. However, Exploring Agent is instead currently providing all direct edges going to a result node in the knowledge graph. It's unclear if one way is recommended over another.

If I understand this correctly, the intent during the design was to provide "paths". The assumption I think is that clients want all the evidence for the results (i.e. "show your work"/"connect the dots") instead of just a final answer. There is also an (unwritten?) assumption that each result should be a complete binding instance of the query_graph. although the notion of incomplete instances is being entertained by the Op&Wf working group.

Perhaps one or more ranks/scores used by the ARA should be provided with each Result. I believe this involves including some of the existing knowledge graph node attributes in the Result. This is to allow the ARS/users to have one place to compare the ranking/scoring of results between ARAs

The extended TRAPI schema has several nice mechanisms for providing scores for results.

edeutsch commented 3 years ago

After discussion at today's call, the consensus seemed to be that this is an issue most relevant to the Operations and Workflows working group and that they are already working to address this. Closing here.

dkoslicki commented 3 years ago

@colleenXu Can you clarify more what your concern is in the first bullet point? I.e. are you saying that Exploring Agent only answers one-hop queries (direct edges going to a result edge)? Or are you saying it takes a path like A->B->C and just returns A->C? We in the Operations and Workflows aren't quite sure what you mean when contrasting "paths" with "direct edges."

colleenXu commented 3 years ago

I believe my first point was based on BTE behavior that we plan to correct (https://github.com/biothings/BioThings_Explorer_TRAPI/issues/164).

Note that my second point seems similar to https://github.com/NCATSTranslator/ReasonerAPI/issues/246 , making it clear what the "score" attribute is.