search

scrapinghub / python-crfsuite

A python binding for crfsuite
MIT License
769 stars 223 forks source link

Can I predict with crf model using multiple processes? I tried, But I got errors -_-! #110

Open huwen2117 opened 4 years ago

huwen2117 commented 4 years ago

I tried, and I got Error: File "stringsource", line 2, in pycrfsuite._pycrfsuite.Tagger.__reduce_cython__ TypeError: self.c_tagger cannot be converted to a Python object for pickling

my plan like this:

model_list = [model_1,model_2, ... ,model_n]

def do_tagging(args):
    model ,sent = args
    tagger = crfs.Tagger()
    tagger.open(model)
    labels = tagger.tag(sent)

with Pool(processes=num_proc) as p:
    predict_result = p.map(do_tagging, zip(model_list, repeat(sent))))
DomHudson commented 4 years ago

Open the model within the process rather than passing the model in.