search

zjwang21 / mix-phoneme-bert

An unofficial PyTorch implementation of Mix-Phoneme-Bert
39 stars 7 forks source link

How to evaluate the model? #4

Open TinaChen95 opened 1 year ago

TinaChen95 commented 1 year ago

Maybe using a fill_mask task to evaluate it ? Or any other suggestions? Could you share some example code of using mix-phoneme mask to do fill_mask task?

I've trained it on 10M chinese sentences, and weirdly get phoneme acc > 1.0. But in your example, it converges to 0.55. Do you have any idea why?

zjwang21 commented 1 year ago

Maybe using a fill_mask task to evaluate it ? Or any other suggestions? Could you share some example code of using mix-phoneme mask to do fill_mask task?

I've trained it on 10M chinese sentences, and weirdly get phoneme acc > 1.0. But in your example, it converges to 0.55. Do you have any idea why?

How do you calculate it? At fairseq/criterions/mix_phoneme_masked_lm.py, I calculate the acc of phoneme and sup-phoneme separately by batch and log them. At line 80-the end. May this can help you.