search

ufal / udpipe

UDPipe: Trainable pipeline for tokenizing, tagging, lemmatizing and parsing Universal Treebanks and other CoNLL-U files
Mozilla Public License 2.0
364 stars 77 forks source link

TypeError: can't pickle udpipe.Model objects #106

Closed ryuzakinho closed 5 years ago

ryuzakinho commented 5 years ago

Hi,

I have been trying to create a copy of a UDPipe Model in Python but I was not able to do so. I am receiving the following error: TypeError: can't pickle udpipe.Model objects.

My Model class is identical to the example in [https://github.com/ufal/udpipe/blob/master/bindings/python/examples/udpipe_model.py](Python bindings example).

To reproduce the error:

import copy  
model = Model(path_to_my_model)
new_model = copy.deepcopy(model)

This example might not seem interesting. The original error started when I tried to use the model with a multiprocessing.Pool. I tried to create one model and use it across different processes (not sure if it is a good idea) but I received the error above. I am now forced to create a new Model instance in each process if I want to avoid the error.

Any recommendations as to how to avoid this error?

Environment:

foxik commented 5 years ago

The Model.model field is a SWIG-wrapped object, which is not pickleable (and I am not planning to allow that). So exactly as you suggest, you must share the path to the model, and then each process has to load its Model. (BTW, even if the model was pickleable, it would do exactly the same, transfer only the path to the model across process boundary, and then load the model again in the new process.)

Closing, but feel free to further comment or reopen.