how to only compute the perplexity of each paragraph using your language model with local data?

facebookresearch / cc_net

Tools to download and cleanup Common Crawl data

MIT License

932 stars 138 forks source link

how to only compute the perplexity of each paragraph using your language model with local data? #54

Open rongjingyue423 opened 11 months ago

rongjingyue423 commented 11 months ago

how to only compute the perplexity of each paragraph using your language model with local data? i don't want to use -d to dump data? I have downloaded the Chinese model use make lang=zh dl_lm

monifeng commented 10 months ago

Hi, did you finish it？I also want to do something like this, maybe we can discuss this and exchange ideas :)