Set num threads ineffective

Tencent / TurboTransformers

a fast and user-friendly runtime for transformer inference (Bert, Albert, GPT2, Decoders, etc) on CPU and GPU.

Other

1.49k stars 198 forks source link

Open auspicious3000 opened 4 years ago

auspicious3000 commented 4 years ago

turbo_transformers.set_num_threads(?)

Whatever number I put in there, it is always using 4 threads.

feifeibear commented 4 years ago

In your command line set export OMP_NUM_THREADS=?

auspicious3000 commented 4 years ago

Does not seem to work either ...

feifeibear commented 4 years ago

If you are using onnxrt as backend. The multiple-threading is not managed by OMP env. export MKL_VERBOSE=1. See the number of threads used for GEMM.