Try to optimize `count_ngrams_in_batches`

shaigue / pmi_masking

This repository contains code that takes a text corpus and creates a PMI masking vocabulary for it.

MIT License

1 stars 0 forks source link

Open shaigue opened 1 year ago

shaigue commented 1 year ago

It might be a good idea to use Numba or other tool to speed-up the relatively simple ngram counting code, if this step appears to be the implementation's bottle neck.