mikeizbicki
commented
11 months ago Based on how the Latin BERT code is written I think I might have to implement the pseudo perplexity myself. This is because the tokenizer they use isn't a derivative of some other more generic tokenizer but instead something that they created which means the generic code I found doesn't work since the tokenizer / model have different outputs / inputs than expected. I think it is possible to get the pseudo perplexity but it might take some time since the Latin BERT code isn't that documented so I first need to understand that. Please let me know if you have any ideas or if I'm missing anything.
This makes sense, but you should probably be able to find a generic pseudocomplexity implementation that lets you specify the tokenizer.
I don't know if this will work for the LatinBERT use case, but I found this SO question with a pseudoperplexity implementation that lets you specify the tokenizer: https://stackoverflow.com/questions/70464428/how-to-calculate-perplexity-of-a-sentence-using-huggingface-masked-language-mode
I think it would potentially be worth looking into taking advantage of the existing features of Latin BERT (ie maybe use text infilling to get the probabilities of the possible words in the sentence and then use that to score based on where the answer is ordered relative to the other words in the word bank, but if not every word in the word bank is present in Latin BERT's dictionary that might require some additional fine tuning / create an issue.
The main disadvantage of this approach is that most words will consist of multiple tokens. (For example the word linguae is likely to be broken down into something like lingu and ae, one token for the root and one for the grammatical ending marker.) So you'll have to account for this, which isn't too bad, but once you are accounting for this, you've basically just implemented the pseudocomplexity yourself.
alysawyer
Hi! Based on how the Latin BERT code is written I think I might have to implement the pseudo perplexity myself. This is because the tokenizer they use isn't a derivative of some other more generic tokenizer but instead something that they created which means the generic code I found doesn't work since the tokenizer / model have different outputs / inputs than expected. I think it is possible to get the pseudo perplexity but it might take some time since the Latin BERT code isn't that documented so I first need to understand that. Please let me know if you have any ideas or if I'm missing anything.
I think it would potentially be worth looking into taking advantage of the existing features of Latin BERT (ie maybe use text infilling to get the probabilities of the possible words in the sentence and then use that to score based on where the answer is ordered relative to the other words in the word bank, but if not every word in the word bank is present in Latin BERT's dictionary that might require some additional fine tuning / create an issue.