Why is the GPU usage low and the CPU usage high when training the model?

[Assassintears]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hello, I train my data with Yolov5.7, it works well. But I find that the GPU usage is very low and the CPU usage high, my train code like bellow:

python train.py --weights xxx --batch_size 8 --imgsz 832 --workers 4 --device 0

Well, if I set the workers parameter to 8 or bigger, the CPU usage may be high to 100%， the training process like bellow:

What should I do to get more high usage of GPU?

Additional

No response

[UltralyticsAssistant]

👋 Hello @Assassintears, thank you for reaching out and sharing your concern with YOLOv5 🚀! This is an automated response, and an Ultralytics engineer will assist you soon.

To address your issue regarding GPU and CPU usage, it's crucial to ensure that your environment is set up optimally. 🛠️ Please provide a minimum reproducible example if possible, as this will greatly aid in diagnosing the issue.

In the meantime, here are some general pointers:

General Tips

1. Workers: Adjusting the --workers parameter can affect how CPU resources are utilized. Finding the right balance based on your hardware is key. You might learn more about this in our Training Tips.
2. Batch size: Try increasing your --batch_size if your GPU memory permits, as this can help improve GPU utilization.

Requirements

Ensure you have Python>=3.8.0 and all necessary packages in requirements.txt installed, including PyTorch>=1.8:

git clone https://github.com/ultralytics/yolov5
cd yolov5
pip install -r requirements.txt

YOLOv5 performs best in environments with the appropriate dependencies including CUDA and CUDNN.

Environments

Choose from several verified environments:

• Notebooks:
• Google Cloud: GCP Quickstart
• Amazon: AWS Quickstart
• Docker: Docker Guide

Status

Check if 🟢 is green, indicating all CI tests are passing.

For a more advanced solution, consider trying out our new YOLOv8 model, with enhanced performance and features:

pip install ultralytics

We appreciate your patience as we work to assist you further! 😊

[michael-mayo]

Try increasing your batch size. With the batch size you are using, GPU utilisation is only 2.79GB. How does performance change if you set the batch size to -1 (automatically set)?

[pderrenger]

Increasing the batch size can help improve GPU utilization. Try setting --batch-size -1 to let the system automatically adjust it based on available memory. Let us know if this improves performance.