nlp.pipe() with multiple processes on Windows VSCode

[IliasAarab]

Trying to run nlp.pipe() with n_process > 1 results in the code hanging indefinitely on my Windows machine when running the code inside a VSCode interactive session. Running the same code from a VSCode Jupyter notebook instance works fine. Running the same code from a traditional Jupyter notebook works fine. Running the same code from a standard python interpreter works fine. Running the same code on Google Colab works fine.

How to reproduce the behaviour

import spacy
nlp = spacy.load("en_core_web_sm")
texts = ["one document to process"]  
results = list(nlp.pipe(texts, n_process=-1))

Your Environment

• spaCy version: 3.7.2
• Platform: Windows-10-10.0.22621-SP0
• Python version: 3.11.7
• Pipelines: de_core_news_sm (3.7.0), en_core_web_sm (3.7.1), es_core_news_sm (3.7.0), fr_core_news_sm (3.7.0), it_core_news_sm (3.7.0)
• VSCode: Version: 1.86.2 (user setup) Commit: 903b1e9d8990623e3d7da1df3d33db3e42d80eda Date: 2024-02-13T19:40:56.878Z Electron: 27.2.3 ElectronBuildId: 26908389 Chromium: 118.0.5993.159 Node.js: 18.17.1 V8: 11.8.172.18-electron.0 OS: Windows_NT x64 10.0.22621

[svlandeg]

Hi! Thanks for the report.

That's weird, I wonder what the difference could be in the VSCode interactive session 🤔

[IliasAarab]

@svlandeg I don't know how nlp.pipe() works under the hood, but I tried executing some basic code in concurrent fashion using ThreadPoolExecutor and this seems to work fine within the interactive session. Let me know if I can provide more information. Would be great if someone can confirm the same issue on their own machine.