support for multiprocessing

nikitakit / self-attentive-parser

High-accuracy NLP parser with models for 11 languages.

https://parser.kitaev.io/

MIT License

871 stars 153 forks source link

support for multiprocessing #12

Closed yangshao closed 5 years ago

yangshao commented 6 years ago

hi, i'm using the cpu version in nltk, i hope to parallel the parser with python's multiprocessing module, but it fails. Is there any way to solve this?

nikitakit commented 6 years ago

The recommended way to parallelize the parser is to use the parse_sents method, which accepts multiple sentences and parses batches of sentences in parallel. This method can utilize all available CPU cores. There is a batch_size argument to the Parser constructor that controls the trade-off between latency and throughput. Does this work for your use case?