deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
https://farm.deepset.ai
Apache License 2.0
1.73k stars 247 forks source link

current process gets forked, after parallelism has already been used #837

Closed EddyDavies closed 2 years ago

EddyDavies commented 3 years ago

Describe the bug When using the DataSilo function following this article the second cell under Data Processor produce this error code. Error message

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)

Expected behavior No error message

To Reproduce Run the Jupyter Notebook from the article

System:

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 21 days if no further activity occurs.