Batch processing on label_mentions is not working

HazyResearch / bootleg

Self-Supervision for Named Entity Disambiguation at the Tail

http://hazyresearch.stanford.edu/bootleg

Apache License 2.0

212 stars 27 forks source link

Batch processing on label_mentions is not working #116

Open coolcoder001 opened 2 years ago

coolcoder001 commented 2 years ago

Hi @lorr1 ,

Is it possible to have batch processing for label_mentions method? Currently it is taking a list of sentences, but internally in the code, I can see it is processing one sentence at a time. Is it possible to do this batchwise to speed up the process?

lorr1 commented 2 years ago

Hey. It would be possible for sure. Setting up some worker queue for the processing.

It's not on my immediate roadmap to solve (as we have offline extract_mentions and dump_preds methods for batch processing). They are in our end2end tutorial.