Training Issues in a Multi-Threaded Environment

scrapinghub / python-crfsuite

A python binding for crfsuite

MIT License

769 stars 223 forks source link

Training Issues in a Multi-Threaded Environment #119

Open peblair opened 4 years ago

peblair commented 4 years ago

We are using this library in a multi-threaded application, and we have run into an issue with thread-friendliness. When the library drops down into C++ and trains the model, it does not appear to be releasing the GIL. This locks up the interpreter (meaning other threads are halted) until training finishes. Would it be possible to tweak the library such that, while executing the crfsuite code, the GIL is released? Thanks!

kmike commented 4 years ago

It should be possible to do so, but I haven't checked it in detail. In the meantime, you can also use processes.