search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.29k stars 21.47k forks source link

Note on synonym order does not accurately describe actual behavior #100620

Closed jesweare closed 1 year ago

jesweare commented 2 years ago

In the Notes section, there is a passage below:

  • Synonyms can be added in any order. The ordering is not considered in any computational logic.

that implies synonym orders would not impact the results of the prediction process.

However, that is not the case; changing the synonym order may create a difference (albeit trivial) in the confidenceScore value, given the Q&A pairs, synonyms, and query are all identical.

The Product Team has confirmed that the score difference is expected, as "computational logic" is representative of only a portion of the whole prediction and ranking process. I assume this rationale can be inferred from the old QnA Maker documentation/diagram explaining the ranker process: How QnA Maker processes a user query to select the best answer

If that is the case, the documentation should add points that makes it clear that score fluctuation may happen. Example below.

  • Synonyms can be added in any order. The ordering is not considered in any computational logic. Please note, however that changing the order may generate negligible differences in the resultant confidence score.

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

AjayBathini-MSFT commented 2 years ago

@jesweare Thanks for your feedback! We will investigate and update as appropriate.

RamanathanChinnappan-MSFT commented 2 years ago

@jesweare

Thanks for your feedback! We have assigned the issue to author and will provide further updates.

jboback commented 1 year ago

Hi @jesweare, the team is aware of this issue and is looking into it. Thank you for reporting the behavior!

jboback commented 1 year ago

please-close