search

pemistahl / lingua-py

The most accurate natural language detection library for Python, suitable for short text and mixed-language text
Apache License 2.0
1.02k stars 43 forks source link

TypeError: cannot pickle `Language` object with v2.0.1 #199

Open tboye opened 7 months ago

tboye commented 7 months ago

After switching to v2.0.1 from v1.3.4 I'm facing TypeError: cannot pickle 'builtins.Language' object when running something along the lines of

import copy
from lingua import Language

l = copy.copy(Language.ENGLISH)

I've checked it with Python 3.10 and Python 3.11.

Am I missing something?

pemistahl commented 7 months ago

Pickling does not work out-of-the-box anymore because you are dealing with a Rust object now, not a Python enum. Pickle support with PyO3 is possible but complicated, that's why I have left it out so far until people start to complain which happened quicker now than I expected.

Why do you need pickle support in the first place? If you are using the multiprocessing module, it is not needed anymore since Lingua now brings its own parallelization methods. If you want to persist Language objects to disk, have you thought about persisting their string representations instead, e.g. Language.ENGLISH.name?

tboye commented 7 months ago

Thanks for the explanation.

I'm using a framework that pickles its settings and one of the settings happened to be a lingua.Language object in my case. But yeah, I can work it around by having a string representation in the settings and then creating an object from it.

Thanks for a nice library!

pemistahl commented 7 months ago

You are welcome. It's good that you have opened this issue. I will try to implement pickle support for the next major release as it is definitely useful to have.

vrdn-23 commented 6 months ago

I would like to see this issue cause we're looking to implement something that doesn't block an async event loop. So we get this same error when trying to put it into a ProcessPoolExecutor. Looking forward to having this ASAP! Thanks a lot and great work!