Provide some ability for running pre-process in parallel

machinalis / iepy

Information Extraction in Python

BSD 3-Clause "New" or "Revised" License

906 stars 186 forks source link

Closed jmansilla closed 9 years ago

jmansilla commented 9 years ago

A simple approach that we tried with success is:

to run preprocess per document, instead of per-step. (Ie, grab a document, run all it's steps, save, grab next one, etc)
create several workers (forking, why not?)
create some policy for distributing documents between them