Intents and Scores in ChildApp do not match the actual LUIS App referenced

dkonyayev commented 4 years ago

Tool

Describe the bug

We are getting incorrect scores for the ChildApp results. For example look at below response. First is the response from LUIS Dispatch, while second is a LUIS query directly against the LUIS App.

Dispatch is correctly identifying the intent with 94% confidence for the utterance. However, the top intent in the ChildApp is "ViewScheduled.PTO" with score of 35%. If we run the same utterance on the app itself we get "ViewAllotment.PTO" with score of 78%.

In our code we adopted the ChildApp attributes to determine the intent so that we do not have to run another LUIS query against the LUIS app. But the results are so drastically different that they become unusable.

Dispatch

{
    "query": "what is my PTO balance?",
    "prediction": {
        "topIntent": "l_app-pto-en-us",
        "intents": {
            "l_app-pto-en-us": {
                "score": 0.9433935,
                "childApp": {
                    "topIntent": "ViewScheduled.PTO",
                    "intents": {
                        "ViewScheduled.PTO": {
                            "score": 0.352805555
                        },
                        "ViewAccrued.PTO": {
                            "score": 0.04455665
                        },
                        "Schedule.PTO": {
                            "score": 0.0265643727
                        },
                        "ViewAvailable.PTO": {
                            "score": 0.0191511624
                        },
                        "Cancel.PTO": {
                            "score": 0.016668899
                        },
                        "ViewTaken.PTO": {
                            "score": 0.008388582
                        },
                        "ViewAllotment.PTO": {
                            "score": 0.004806818
                        },
                        "None": {
                            "score": 0.00119963824
                        }
                    },
                    "entities": {
                        "keyPhrase": [
                            "PTO balance"
                        ],
                        "$instance": {
                            "keyPhrase": [
                                {
                                    "type": "builtin.keyPhrase",
                                    "text": "PTO balance",
                                    "startIndex": 11,
                                    "length": 11,
                                    "modelTypeId": 2,
                                    "modelType": "Prebuilt Entity Extractor",
                                    "recognitionSources": [
                                        "model"
                                    ]
                                }
                            ]
                        }
                    },
                    "sentiment": {
                        "label": "negative",
                        "score": 0.261277676
                    }
                }
            },
            "q_pto-en-us": {
                "score": 0.06487191
            },
            "None": {
                "score": 9.814609E-05
            }
        },
        "entities": {},
        "sentiment": {
            "label": "negative",
            "score": 0.261277676
        }
    }
}

App

{
    "query": "what is my PTO balance?",
    "prediction": {
        "topIntent": "ViewAllotment.PTO",
        "intents": {
            "ViewAllotment.PTO": {
                "score": 0.779008865
            },
            "ViewAvailable.PTO": {
                "score": 0.0923415348
            },
            "ViewAccrued.PTO": {
                "score": 0.0389702581
            },
            "ViewScheduled.PTO": {
                "score": 0.008471076
            },
            "Schedule.PTO": {
                "score": 0.00402491447
            },
            "Cancel.PTO": {
                "score": 0.00312482542
            },
            "None": {
                "score": 0.0007331176
            },
            "ViewTaken.PTO": {
                "score": 0.0006362235
            }
        },
        "entities": {
            "keyPhrase": [
                "PTO balance"
            ],
            "$instance": {
                "keyPhrase": [
                    {
                        "type": "builtin.keyPhrase",
                        "text": "PTO balance",
                        "startIndex": 11,
                        "length": 11,
                        "modelTypeId": 2,
                        "modelType": "Prebuilt Entity Extractor",
                        "recognitionSources": [
                            "model"
                        ]
                    }
                ]
            }
        },
        "sentiment": {
            "label": "negative",
            "score": 0.261277676
        }
    }
}

To Reproduce

Run the query against dispatch and app and compare results.

Expected behavior

We expected the intents and scores under ChildApp to match what we would get if we ran the query directly against the LUIS App.

[bug]

tsuwandy commented 4 years ago

@dkonyayev , how do you set the "childApp" attribute? I thought in combined queries, it's called "connectedServiceResult"

dkonyayev commented 4 years ago

@tsuwandy I didn't set it up. I think it started to appear after Luis v3 was introduced. There was nothing that we did to "enable" it.

tsuwandy commented 4 years ago

Could you check the app settings on both dispatch and child LUIS apps? I’ve seen this happened before and the difference in settings between dispatch and child LUIS app were to blame. You could use the luis API console to get app settings if they are not displayed in luis portal.

Let me know if you see differences in settings. Otherwise this would be something LUIS support should investigate. Thanks!

dkonyayev commented 4 years ago

I don’t see anything different in the portal. What is LUIS API Console? I am not familiar with this tool and not sure where to get it.

Thanks,

Dmitriy Konyayev +1 813-682-9607 dkonyayev@hotmail.commailto:dkonyayev@hotmail.com https://www.linkedin.com/in/dmitriy-konyayev

From: Tien Suwandy notifications@github.com Sent: Wednesday, August 5, 2020 9:31 PM To: microsoft/botbuilder-tools botbuilder-tools@noreply.github.com Cc: dkonyayev dkonyayev@hotmail.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-tools] Intents and Scores in ChildApp do not match the actual LUIS App referenced (#1464)

Could you check the app settings on both dispatch and child LUIS apps? I’ve seen this happened before and the difference in settings between dispatch and child LUIS app were to blame. You could use the luis API console to get app settings if they are not displayed in luis portal.

Let me know if you see differences in settings. Otherwise this would be something LUIS support should investigate. Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fbotbuilder-tools%2Fissues%2F1464%23issuecomment-669630195&data=02%7C01%7C%7C7f4935a7c5ff4a902e2908d839a85bb8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637322742532337327&sdata=hubn1GNVpF%2B911BjNdWZo0SkIjwcauGU2PeyNUwuKrw%3D&reserved=0, or unsubscribehttps://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FADVCG4UPLZIGFI37THDBB2LR7IBUZANCNFSM4PVTRIZA&data=02%7C01%7C%7C7f4935a7c5ff4a902e2908d839a85bb8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637322742532347322&sdata=uZIHPNcOuhPPmpyQyDwFil9Yjk58SxdYW6v6Q4TFUiU%3D&reserved=0.

tsuwandy commented 4 years ago

@dkonyayev , here's the console. Let me know if you see differences in the settings of Dispatch app version and child LUIS app version. If you do, we'll need to raise this issue to LUIS support. Thanks!

https://westus.dev.cognitive.microsoft.com/docs/services/luis-programmatic-apis-v3-0-preview/operations/versions-get-application-version-settings

peterinnesmsft commented 4 years ago

Closing due to further inactivity. Feel free to reactivate if further assistance is required.

microsoft / botbuilder-tools