Open arnicas opened 4 years ago
Oops. The current draft seems to be missing a link to where we got the wordlists from: https://www.enchantedlearning.com/wordlist/. Will add this back into the paper! Thanks for catching this.
Aside: Right now, the code only allows for words that are 1 BPE token long. Handling multiple tokens would need a few minor changes.
Thanks for the suggestion; yes, I agree it would be a good idea to make it easier to use with your own BoW. Will consider incorporating this!
So are you saying that all words in a wordlist have to be 1 word only? Not carpenter ant
?
Hi, this looks great. I had to look at the code to get some insight into how to do a BOW approach of my own. Maybe you could add a few lines to the readme about that? The paper seems a little light on how the topic words were selected as well, unless I missed that? But awesome work!