IntentRecognizer Set not working for multiple recognizers

The Microsoft Bot Framework team prefers that "How To" questions be submitted on Stack Overflow. The official Bot Framework GitHub repository  is the preferred platform for submitting bug fixes and feature requests.

Bot Info

SDK Platform:Nodejs
SDK Version: 3.15.0
Active Channels: web
Deployment Environment: emulator

Issue Description

Issue . RecognizerSet was created with two LuisRecognizers. It works well with one recgonizer. If I create with two recorngizers the second one only is used. Tried with parallel, series etc. Please note, I am not using IntentDialog. I am using IntentRecognizerSet.recognize

Code Example

luisApps = [{id:'', key:''}, {id:'', key:''}]
let prepareLUISURL = = (id, key) => {
   return `https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/${id}?verbose=true&timezoneOffset=0&subscription-key=${key}`
}
luisApps.forEach((a) => {
            recognizers.push(new builder.LuisRecognizer(prepareLUISUrl(a.id, a.key)))
        })
let rSet = new builder.IntentRecognizerSet({
            recognizers: recognizers
        })
rSet.recognize(session, (e, r) => {
            if (e) {
                if (cb) cb(e)
                return null
            } else {
              cb(null,r)
        }   
})

Reproduction Steps

as indicated above.

Expected Behavior

intents from both the models.

Actual Results

Only intents of the second recognizer are returned.

@ubreddy could you please share the code of what your "cb" is? Trying to repro your results to drill down further into the issue :)

@Zerryth : cb is just a callback function to collect the results... it could be even console.log instead

rSet.recognize(session, (e, r) => {
            if (e) {
                console.log(e)
                return null
            } else {
              console.log(r)
        }   
})

@ubreddy Thanks for the response.

I was unable to reproduce the error you had gotten and used the following code as a working solution for the use of IntentRecognizerSet:

Adding recognizers

const recognizers = [];

const luisAPIKey = process.env.LuisAPIKey;
const notesLuisAppId = process.env.NotesLuisAppId;
const homeAutomationLuisAppId = process.env.HomeAutomationLuisAppId;
const luisAPIHostName = process.env.LuisAPIHostName || 'westus.api.cognitive.microsoft.com';

let prepareLuisUrl = (id, key) => {
    return `https://${luisAPIHostName}/luis/v2.0/apps/${id}?verbose=true&timezoneOffset=0&subscription-key=${key}`;
 }

const luisApps = [ 
    {id: notesLuisAppId, key: luisAPIKey},
    {id: homeAutomationLuisAppId, key: luisAPIKey}
];

luisApps.forEach((application) => {
    recognizers.push( new builder.LuisRecognizer(prepareLuisUrl(application.id, application.key)) );
    console.log(recognizers);
});
let rSet = new builder.IntentRecognizerSet({
    recognizers: recognizers
});
rSet.recognize(session, (err, result) => {
    if (err) {
        console.log(err);
    } else {
        console.log(`rSet.recognize results`);
    }
});

for (let createdRecognizer of recognizers)
{
    bot.recognizer(createdRecognizer);
}

The main differences being that I do not return null if there's an err and I also make sure to loop through all recognizers to add each created recognizer to the universal bot.

It's not posted in your code snippet, but I would check to see if you're bot code makes sure to add not only 1 recognizer but all of them that you create to your bot

Edit: adding bot dialog and chat results screenshot

Bot Dialog - Matching to Intents

bot.dialog('CreateNoteDialog',
    (session) => {
        session.send('You reached the Create Note intent!! You said \'%s\'.', session.message.text);
        session.endDialog();
    }
).triggerAction({
    matches: 'Note.Create'
})

bot.dialog('TurnOnLights',
    (session) => {
        session.send('You reached Turn On Lights intent!! You said \'%s\'.', session.message.text);
        session.endDialog();
    }
).triggerAction({
    matches: 'HomeAutomation.TurnOn'
})

Successfully recognizes intents from two separate LUIS apps

createnoteandturnonlights

@Zerryth May be it is coming inside bot as you are adding each recognizer to bot. Please print the results in rSet.recognize funciton... console.log('rSet.recognize results', result); The result here is not showing all intents. and check the result there... bot.recognize seems fine. rSet.recognize is not.

Context:

I am not using bot recognizers, but dynamically switching recognizers based on user attributes... so, cannot preload recognizers with bot.. hence using rSet in middleware...

Solution, is to externalize recognizers and delink to bot. but the bug is that rSet.recognize is not giving its output....

Ok, looks like what want to use is .recognizeInSeries

Right now I've just hard-coded my context to be a message that says 'turn on the light'. Luis recognizes it as the HomeAutomation.TurnOn intent, and it works with rSet

.recognizeInSeries

let prepareLuisUrl = (id, key) => {
    return `https://${luisAPIHostName}/luis/v2.0/apps/${id}?verbose=true&timezoneOffset=0&subscription-key=${key}`;
 }

const luisApps = [ 
    {id: notesLuisAppId, key: luisAPIKey},
    {id: homeAutomationLuisAppId, key: luisAPIKey}
];

luisApps.forEach((application) => {
    recognizers.push( new builder.LuisRecognizer(prepareLuisUrl(application.id, application.key)) );
    console.log(recognizers);
});
const rSet = new builder.IntentRecognizerSet({
    recognizers: recognizers
});
rSet.recognizeInSeries(
    { message: {
        text: 'turn on the light'
    }}, 
    (err, result) => {
    if (err) {
        console.log(err);
        return null;
    } else {
        console.log('L85: recognizeInSeries')
        console.log(`rSet.recognize results: *******${JSON.stringify(result)}**********`);
    }
});

bot.recognizer(rSet);

results:

L85: recognizeInSeries
rSet.recognize results: 
*******
{
    "score":0.985385954,
    "intent":"HomeAutomation.TurnOn",
    "intents":[
        {
            "intent":"HomeAutomation.TurnOn",
            "score":0.985385954
        },
        {
            "intent":"HomeAutomation.TurnOff",
            "score":0.0298946556
        },
        {
            "intent":"None",
            "score":0.0135929184
        }
    ],
    "entities":[
        {
            "entity":"light",
            "type":"HomeAutomation.Device",
            "startIndex":12,
            "endIndex":16,
            "score":0.9716048
        }
    ],
    "compositeEntities":[]
}
**********

Yes. I want both the models intents to be recognized and scored. Not only one model... the problem I am reporting is if there are two luis models, the message should be matched against the both models and report the intents ranked by score from both models.

Not stop with the first model alone.... How do I achieve this? I have used the recognizeOrder options too as parallel (the default), but it does not seem to work

@ubreddy

I think fundamentally there's a misunderstanding of what the bot framework is supposed to do vs. what you expect it to do.

What actually happens with bot framework

Given the following code, using either rSet.recognize() or rSet.recognizeInSeries(),

IntentRecognizerSet will take in a message (utterance)
check the utterance against all the LUIS models in the recognizers (which for me in my case I have a model that has home automation intents and a second model that has create notes intents)
as rSet is going through and checking the scores returned against each model, it will only return the intent with the highest score in the results
- "results" was not programmed in the framework to return all intents for every single model,
- but instead it iterates through all the models, and keeps the results of the intent with the highest score, regardless of which model it is using

You can find the full code of this functionality in the framework within the IntentRecognizerSet class file

private recognizeInSeries(context: IRecognizeContext, done: (err: Error, result: IIntentRecognizerResult) => void): void {
        var i = 0;
        var result: IIntentRecognizerResult = { score: 0.0, intent: null };
        async.whilst(() => {
            return (i < this.options.recognizers.length && (result.score < 1.0 || !this.options.stopIfExactMatch));
        }, (cb) => {
            try {
                var recognizer = this.options.recognizers[i++];
                recognizer.recognize(context, (err, r) => {
                    // HERE DESCRIBES THE BEHAVIOR I OUTLINED ABOVE IN BULLETS
                    if (!err && r && r.score > result.score && r.score >= this.options.intentThreshold) {
                        // YOUR RESULT WILL ONLY HAVE THE LUIS RESULT OF HIGHEST-SCORING INTENT
                        result = r;
                    }
                    cb(err);
                });
            }

So when I use the following code, below, I send in an utterance, rSet checks both of my 2 models, returns the result from the model with the highest-scoring intent:

const recognizers = [];

const luisAPIKey = process.env.LuisAPIKey;
const notesLuisAppId = process.env.NotesLuisAppId;
const homeAutomationLuisAppId = process.env.HomeAutomationLuisAppId;
const luisAPIHostName = process.env.LuisAPIHostName || 'westus.api.cognitive.microsoft.com';

let prepareLuisUrl = (id, key) => {
    return `https://${luisAPIHostName}/luis/v2.0/apps/${id}?verbose=true&timezoneOffset=0&subscription-key=${key}`;
 }

const luisApps = [ 
    {id: notesLuisAppId, key: luisAPIKey},
    {id: homeAutomationLuisAppId, key: luisAPIKey}
];

luisApps.forEach((application) => {
    recognizers.push( new builder.LuisRecognizer(prepareLuisUrl(application.id, application.key)) );
});

const rSet = new builder.IntentRecognizerSet({
    recognizers: recognizers
});
 rSet.recognizeInSeries(
//rSet.recognize(
    { message: {
        text: 'create a note'
    }},
    (err, result) => {
    if (err) {
        console.log(err);
        return null;
    } else {
        console.log('L85: recognizeInSeries')
        console.log(`rSet.recognize results: *******${JSON.stringify(result)}**********`);
    }
});

Results:

L85: recognizeInSeries
rSet.recognize results: 
*******{
    "score":0.9039253,
    "intent":"Note.Create",
    "intents":[
        {"intent":"Note.Create","score":0.9039253},
        {"intent":"NoteLights","score":0.04346403},
        {"intent":"None","score":0.0307157263},
        {"intent":"Greeting","score":0.008593935},
        {"intent":"Note.Delete","score":0.003751613},
        {"intent":"Cancel","score":0.00214334484},
        {"intent":"Help","score":0.00115081831},
        {"intent":"Note.ReadAloud","score":0.000804136565}
    ],
    "entities":[],
    "compositeEntities":[]
}**********

What you need to do to achieve your desired solution

So, if I'm now understanding you correctly, you actually want the results of all models to return, not just the results of the model with the top-scoring intent.

To do this, you would simply have to edit one of the bot framework files directly, with customization to how you'd like the results to appear.

For example, changing line 125 of IntentRecognizerSet's .recognizeInSeries() method Right now, thrown together very quickly I've edited the code to not have the condition to look if the results of the LUIS model that IntentRecognizerSet is currently at out-scores the previously-looked-at models (i.e. omitting r.score > result.score in the condition)

(in my own code downloaded I have IntentRecognizerSet.js instead of .ts)

IntentRecognizerSet.js, Line 88

IntentRecognizerSet.prototype.recognizeInSeries = function (context, done) {
        var _this = this;
        var i = 0;
        var result = { score: 0.0, intent: null};
        async.whilst(function () {
            return (i < _this.options.recognizers.length && (result.score < 1.0 || !_this.options.stopIfExactMatch));
        }, function (cb) {
            try {
                var recognizer = _this.options.recognizers[i++];
                recognizer.recognize(context, function (err, r) {
                    if (!err && r &&  r.score >= _this.options.intentThreshold) {
                        if(result.score != 0.0){

                            result["otherIntents"].push(r);
                        }   
                        else{
                            result = r;
                            // creating another property in results called "otherIntents"
                            // that will be an array of intents from models that scored lower than the highest-scoring model
                            result["otherIntents"] =[];
                        }

                    }
                    cb(err);
                });
            }
            catch (e) {
                cb(e);
            }
        }, function (err) {
            if (!err) {
                done(null, result);
            }
            else {
                done(err instanceof Error ? err : new Error(err.toString()), null);
            }
        });

Results from customized edits to the framework directly

{
    "score":0.9039253,
    "intent":"Note.Create",
    "intents":[
        {"intent":"Note.Create","score":0.9039253},
        {"intent":"NoteLights","score":0.04346403},
        {"intent":"None","score":0.0307157263},
        {"intent":"Greeting","score":0.008593935},
        {"intent":"Note.Delete","score":0.003751613},
        {"intent":"Cancel","score":0.00214334484},
        {"intent":"Help","score":0.00115081831},
        {"intent":"Note.ReadAloud","score":0.000804136565}
    ],
    "entities":[],
    "compositeEntities":[],
    "otherIntents":[
        {
            "score":0.224398091,
            "intent":"HomeAutomation.TurnOn",
            "intents":[
                {"intent":"HomeAutomation.TurnOn","score":0.224398091},
                {"intent":"None","score":0.05094016},
                {"intent":"HomeAutomation.TurnOff","score":0.0491067544}
            ],
            "entities":[],
            "compositeEntities":[]
        }
    ]
}

Closing issue on GitHub, as this is not a bug. If you'd like assistance of customizing the logic of the framework to suit your needs specifically, please ask the community on StackOverflow

microsoft / botframework-sdk