search

k2-fsa / icefall

https://k2-fsa.github.io/icefall/
Apache License 2.0
885 stars 286 forks source link

Zipformer streaming model: repeated single-syllable words/characters stop abruptly #1356

Open rouseabout opened 10 months ago

rouseabout commented 10 months ago

I have been experimenting with the pre-trained zipformer streaming models:

A problem I observe is the models do not recognise more than two single-syllable words/characters in a sequence. The recogniser stop emitting tokens after the first two words or characters.

For example:

The problem goes away if single syllable words/characters are mixed with other single syllable words/characters. For example:

rouseabout commented 10 months ago

Problem also occurs with non-streaming models, e.g.:

Given this happens across different datasets and training runs, is there something about the zipformer model architecture that intentionally limits output of many identical single-syllable word/character tokens?

audio-samples-en-zh.zip

joazoa commented 7 months ago

I also experience this issue.

danpovey commented 7 months ago

Can it relate to the stateless transducer context length of 2? It doesn't know how many copies of the symbol it has already emitted and guesses it has already output them. At one point we experimented with augmenting the context with a repeat-count, but we didn't end up merging it.

On Wed, Jan 17, 2024, 4:53 AM joazoa @.***> wrote:

I also experience this issue.

— Reply to this email directly, view it on GitHub https://github.com/k2-fsa/icefall/issues/1356#issuecomment-1894495177, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAZFLO7XCV55ES7HDDPHJBDYO3SCZAVCNFSM6AAAAAA6XDOPLGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJUGQ4TKMJXG4 . You are receiving this because you are subscribed to this thread.Message ID: @.***>