search

utterworks / fast-bert

Super easy library for BERT based NLP models
Apache License 2.0
1.86k stars 341 forks source link

enhancement: creating tokenised features in batches #192

Open jkhalsa-arabesque opened 4 years ago

jkhalsa-arabesque commented 4 years ago

it seems that creating tokenised features is a very memory intensive task, perhaps by doing this in batches it can alleviate the RAM usage a little.