Describe the bug
When using the DataSilo function following this article the second cell under Data Processor produce this error code.
Error message
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Expected behavior
No error message
To Reproduce
Run the Jupyter Notebook from the article
Describe the bug When using the DataSilo function following this article the second cell under Data Processor produce this error code. Error message
Expected behavior No error message
To Reproduce Run the Jupyter Notebook from the article
System: