search

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
https://farm.deepset.ai
Apache License 2.0
1.74k stars 248 forks source link

GPU usage very low in inferences #752

Closed bipinkc19 closed 3 years ago

bipinkc19 commented 3 years ago

Question Farm when use_gpu=True is using very low gpu. How do I up this? \

Timoeller commented 3 years ago

Hey, this depends a lot on the data you put into the system. Can you give some more details on this data.

Also inference often does not fully leverage the GPU RAM, since communication between CPU and GPU is the bottleneck there.

In the meantime you can set batch_size=40 and num_processes=1, this should improve GPU usage for most cases.

Timoeller commented 3 years ago

Issue seems resolved, closing now, feel free to reopen