进行评测时会报错,分叉可能会导致死锁.

[TC9905]

🐛 bug 说明

huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either:

• Avoid using tokenizers before the fork if possible
• Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) 报错信息如上.

Python Version

None

[wangyuxinwhy]

谢谢反馈，这看起来是 tokenizers 的问题，需要设置一下才行，我去读读文档看看要不要在脚本中自动添加这个环境变量。