huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

https://huggingface.co/transformers

Apache License 2.0

132.7k stars 26.44k forks source link

add preprocessing_num_workers to run_classification.py #31586

Closed jiahuanluo closed 3 months ago

jiahuanluo commented 3 months ago

preprocessing_num_workers option to speed up preprocess

What does this PR do?

Add preprocessing_num_workers to speedup tokenizer.

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR. @ArthurZucker