multiprocessing: Can't pickle local object 'build_ngram_suggester.<locals>.ngram_suggester

[vitaly-d]

spacy.ngram_suggester.v1 and spacy.ngram_range_suggester.v1 suggesters don't seem to be compatible with multiprocessing: it fails withAttributeError: Can't pickle local object 'build_ngram_suggester.<locals>.ngram_suggester' .

How to reproduce the behaviour

the spancat_multiprocessing_fail.py script from the attached spancat_multiprocessing.zip fails with AttributeError: Can't pickle local object 'build_ngram_suggester.<locals>.ngram_suggester'

Workaround

the spancat_multiprocessing_ok.py script works. It contains 'custom' suggester that is a copy of https://github.com/explosion/spaCy/blob/372a90885e849a286c27103bece93ad8a02c842e/spacy/pipeline/spancat.py#L86 wrapped in a python class.

Info about spaCy

• spaCy version: 3.5.0
• Platform: macOS-12.6.4-arm64-arm-64bit
• Python version: 3.10.6
• Pipelines: en_citations_parser (0.15.0), en_core_web_sm (3.5.0)

[adrianeboyd]

Thanks for the report, I'm surprised this hasn't come up sooner!

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.