Kenlm doesn't works with Batch Inference

AI4Bharat / IndicWav2Vec

Pretraining, fine-tuning and evaluation scripts for Indic-Wav2Vec2

https://indicnlp.ai4bharat.org/indicwav2vec

MIT License

82 stars 28 forks source link

Kenlm doesn't works with Batch Inference #6

Closed harveenchadha closed 2 years ago

harveenchadha commented 2 years ago

Hello,

I tried the single file inference, this worked with both viterbi and kenlm. Then I moved to batch inference, here viterbi works just fine but kenlm throws an exception:

Normally this exception comes when the vocab of the model and the language model is different but I think this would be taken care of for sure. Can you suggest something to run this? I am also surprised as I am not getting this error in single file inference but in batch mode.

Thanks!

tahirjmakhdoomi commented 2 years ago

I see. As a quick fix can you try commenting out the assert statement in w2l_decoder (inside init,W2lKenLMDecoder) and see if that works?

harveenchadha commented 2 years ago

Even in my wildest dreams, I would not have imagined that this would work but it did. Looks like assert is doing nothing 😃

Thank you.