Excessive CPU usage - Yolov5 Inference on Mac M1

asriaws commented 1 year ago

Search before asking

[X] I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

No response

Bug

I am using live object detection using Logitect web camera on my Mac M1 machine with 16GB memory with Yolov5 model. Here are my findings Yolov5n: Average CPU utilization 140% Yolov5s: Average CPU utilization 185% Yolov5m: Average CPU utilization 214% So far inference is running without any issue, but Mac is overheating. Is there a way to reduce overall CPU utilization?

Environment

Yolov5 Mac M1

Minimal Reproducible Example

python detect.py --weights playingcardsm.pt --source 0

Additional

No response

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

github-actions[bot] commented 1 year ago

👋 Hello @asriaws, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

glenn-jocher commented 1 year ago

@asriaws if you are running on a mac you may get better performance with --device mps, or even better you should export to CoreML and then run inference with your CoreML model directly, it will be about 10X faster than your PyTorch model.

asriaws commented 1 year ago

Thanks @glenn-jocher. I am getting amazing performance with current setup. Each inference runs at 100 ms. The challenge is more towards high CPU utilization. Do you think still the options suggested would work?

glenn-jocher commented 1 year ago

@asriaws high CPU utilization can be a common issue when running YOLOv5 on certain hardware configurations. While using the --device mps flag may help alleviate the issue to some extent, it's worth noting that the most efficient way to run YOLOv5 on a Mac M1 is by exporting your model to CoreML format and running inference using CoreML. This can significantly reduce CPU utilization and provide a faster inference speed, roughly 10 times faster compared to running it with PyTorch. I would recommend giving this approach a try to see if it helps with the high CPU utilization you're experiencing. Let me know if you have any further questions or need assistance with the export process.

asriaws commented 1 year ago

Thanks @glenn-jocher. I ran detect.py with --device mps and it brought down CPU utilization from 214% to 56%, which is exciting. I am trying to CoreML option but getting some error . Can you help?

I am using python virtual env on Mac.
Python version 3.11.0
Used python export.py --weights yolo5s.pt --include "coreml" to convert the PyTorch model to CoreML
Istalled pip3 install coremltools
Ran following command python detect.py --weights yolov5s.mlmodel --source 0 --device mps

(also tried to downgrade PyTorch and TensorFlow version but no luck) Got following error


Loading yolov5s.mlmodel for CoreML inference...
TensorFlow version 2.13.0 has not been tested with coremltools. You may run into unexpected errors. TensorFlow 2.12.0 is the most recent version that has been tested.
Torch version 2.0.1 has not been tested with coremltools. You may run into unexpected errors. Torch 2.0.0 is the most recent version that has been tested.
1/1: 0...  Success (inf frames 1920x1080 at 5.00 FPS)

Exception loading model proxy: No module named 'coremltools.libcoremlpython'

Traceback (most recent call last): File "/Users/asriaws/pokerbot/yolov5/detect.py", line 262, in main(opt) File "/Users/asriaws/pokerbot/yolov5/detect.py", line 257, in main run(vars(opt)) File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/yolov5/detect.py", line 129, in run pred = model(im, augment=augment, visualize=visualize) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/yolov5/models/common.py", line 545, in forward y = self.model.predict({'image': im}) # coordinates are xywh normalized ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/coremltools/models/model.py", line 535, in predict raise Exception("Unable to load CoreML.framework. Cannot make predictions.") Exception: Unable to load CoreML.framework. Cannot make predictions.

glenn-jocher commented 1 year ago

@asriaws, glad to hear that using --device mps reduced the CPU utilization for you! Regarding the error you encountered when trying to run YOLOv5 with CoreML, it seems like there might be an issue with the installation or compatibility of coremltools.

To resolve this, I recommend checking your coremltools installation and ensuring that it is up-to-date. You can use the command pip show coremltools to check the version. If it's not the latest version, you can update it using pip install --upgrade coremltools.

Additionally, please make sure that you have the necessary dependencies installed for CoreML. You may want to refer to the official CoreML documentation for the required dependencies and installation instructions.

If the issue persists, you can try uninstalling and reinstalling coremltools to ensure a clean installation. You can use pip uninstall coremltools to remove it and then reinstall it with pip install coremltools.

Lastly, please ensure that you are using compatible versions of TensorFlow and PyTorch as mentioned in the error message. You may want to consider using the tested versions (TensorFlow 2.12.0 and Torch 2.0.0) to avoid any unexpected errors.

I hope this helps resolve the issue. Let me know if you have any further questions or need additional assistance!

asriaws commented 1 year ago

@glenn-jocher, I tried couple of option but no luck. Please let me know if you can suggestion something else.

Downgrade TensorFlow to 2.12.0 and Torch to 2.0.0. Now the warning is gone but same issue reappeared.
Uninstalled coremltools and reinstalled back
Deleted and recreated new version of yolov5s.mlmodel
pip show coremltools: Version: 6.3.0 is latest version

Following command and error


(env) $python detect.py --weights yolov5m.mlmodel --source 0         
detect: weights=['yolov5m.mlmodel'], source=0, data=data/coco128.yaml, imgsz=[640, 640], conf_thres=0.25, iou_thres=0.45, max_det=1000, device=, view_img=False, save_txt=False, save_conf=False, save_crop=False, nosave=False, classes=None, agnostic_nms=False, augment=False, visualize=False, update=False, project=runs/detect, name=exp, exist_ok=False, line_thickness=3, hide_labels=False, hide_conf=False, half=False, dnn=False, vid_stride=1
YOLOv5 🚀 v7.0-204-g6ec2bfe Python-3.11.0 torch-2.0.0 CPU

Loading yolov5m.mlmodel for CoreML inference... 1/1: 0... Success (inf frames 1920x1080 at 30.00 FPS)

Exception loading model proxy: No module named 'coremltools.libcoremlpython'

Traceback (most recent call last): File "/Users/asriaws/pokerbot/yolov5/detect.py", line 262, in main(opt) File "/Users/asriaws/pokerbot/yolov5/detect.py", line 257, in main run(vars(opt)) File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/yolov5/detect.py", line 129, in run pred = model(im, augment=augment, visualize=visualize) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/yolov5/models/common.py", line 545, in forward y = self.model.predict({'image': im}) # coordinates are xywh normalized ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/asriaws/pokerbot/env/lib/python3.11/site-packages/coremltools/models/model.py", line 535, in predict raise Exception("Unable to load CoreML.framework. Cannot make predictions.") Exception: Unable to load CoreML.framework. Cannot make predictions.

glenn-jocher commented 1 year ago

@asriaws i understand that you have been experiencing difficulties with running YOLOv5 using CoreML. You have tried several options, including downgrading TensorFlow and Torch, reinstalling coremltools, and recreating the yolov5s.mlmodel file. However, despite your efforts, you are still encountering an error.

Based on the error message you provided, it appears that there is an issue with loading the CoreML framework, leading to the inability to make predictions with the model.

To address this issue, I would recommend the following steps:

Double-check that you have installed all the necessary dependencies for CoreML and that they are up-to-date. Ensure that you have installed CoreML using pip install coremltools and that you have the latest version, which is currently 6.3.0.
Verify that you have the correct versions of TensorFlow and Torch. As mentioned in the error message, TensorFlow 2.12.0 and Torch 2.0.0 are the most recent versions that have been tested with coremltools. Make sure you have these versions installed.

If the issue persists after following these steps, it could be related to your specific environment or setup. In such cases, I would recommend seeking further assistance from the YOLOv5 GitHub community or the official CoreML documentation, as they may have more specific insights or solutions.

Please let me know if you have any further questions or if there is anything else I can assist you with.

github-actions[bot] commented 1 year ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

ultralytics / yolov5