Implementing contextual biasing

[mlecauchois]

Hello!

Thanks to k2 I have successfully used wfst ctc decoding. I am now trying to understand how to go about implementing contextual biasing. That is, given my main LM G, I wish to bias it offline with a second small contextual LM B using log-linear interpolation to achieve something like equation (1) of Bringing Contextual Information to Google Speech Recognition.

Any ideas on how I should start working on this ?

[danpovey]

If your biasing is just per word, you could just do something like: G = G.clone() G.scores += bias_vec[G.labels]

On Fri, Aug 13, 2021 at 4:24 AM mlecauchois @.***> wrote:

Hello!

Thanks to k2 I have successfully used wfst ctc decoding. I am now trying to understand how to go about implementing contextual biasing. That is, given my main LM G, I wish to bias it offline with a second small contextual LM B using log-linear interpolation to achieve something like equation (1) of Bringing Contextual Information to Google Speech Recognition https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43819.pdf .

Any ideas on how I should start working on this ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO24IWXKQ7XAEC6752LT4QUYNANCNFSM5CCCUJWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[mlecauchois]

Thanks! This indeed looks promising. For broader biasing, with a second n-gram LM for instance, what should I look into ? Are there general approaches (that could leverage k2) to interpolating such WFSTs ?

[danpovey]

You could just intersect it with your biasing FSA. Make sure your biasing FSA has weights which can be interpreted as corrections, i.e. new/old.

On Fri, Aug 13, 2021 at 5:17 PM mlecauchois @.***> wrote:

Thanks! This indeed looks promising. For broader biasing, with a second n-gram LM for instance, what should I look into ? Are there general approaches to interpolating such WFSTs ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/245#issuecomment-898311192, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLOZISXWD2DJOKEPOPATT4TPJHANCNFSM5CCCUJWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[danpovey]

... actually new minus old, since the scores are in log-space.

On Fri, Aug 13, 2021 at 6:29 PM Daniel Povey @.***> wrote:

You could just intersect it with your biasing FSA. Make sure your biasing FSA has weights which can be interpreted as corrections, i.e. new/old.

On Fri, Aug 13, 2021 at 5:17 PM mlecauchois @.***> wrote:

Thanks! This indeed looks promising. For broader biasing, with a second n-gram LM for instance, what should I look into ? Are there general approaches to interpolating such WFSTs ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/245#issuecomment-898311192, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLOZISXWD2DJOKEPOPATT4TPJHANCNFSM5CCCUJWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[mlecauchois]

Ok this makes sense thanks!

Since the biasing LM has a potentially smaller span than the main LM, intersection could suppress a lot of the arcs of the main LM that would still be relevant for decoding (instead the biasing LM should bias a small subset of arcs, while leaving the others untouched and available for decoding).

To solve that issue, I suppose that an alternative would be to add self-loops with the whole lexicon as labels to the first state of the biasing LM. That way, the intersection leaves the original main LM intact, but biases the relevant subset of arcs contained in the biasing LM right? Of course, since the original arcs are still there and we use the path with max weight, that works only if the biasing LM increases the weight of an arc.

[danpovey]

Ok this makes sense thanks!

Since the biasing LM has a potentially smaller span than the main LM, intersection could suppress a lot of the arcs of the main LM that would still be relevant for decoding (instead the biasing LM should bias a small subset of arcs, while leaving the others untouched and available for decoding).

To solve that issue, I suppose that an alternative would be to add self-loops with the whole lexicon as labels to the first state of the biasing LM. That way, the intersection leaves the original main LM intact, but biases the relevant subset of arcs contained in the biasing LM right? Of course, since the original arcs are still there and we use the path with max weight, that works only if the biasing LM increases the weight of an arc.

Mm, I was kind of assuming all the words were there in the biasing LM (just with different probability), and that it had backoff arcs (via epsilon). If you want to run intersection on GPU you'd have to add epsilon self-loops to the main LM, so that the epsilons will match.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/k2-fsa/snowfall/issues/245#issuecomment-898613878, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO5MUH3SRO3LZ6LLAFLT4VJCFANCNFSM5CCCUJWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[mlecauchois]

Alright that did work out, thanks a lot for the help! I am closing the issue