WARNING ⚠️ NMS time limit 2.100s exceeded

sanjanak-rbg commented 11 months ago

Search before asking

[X] I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Validation

Bug

3/15 4.89G 0.5933 0.5576 1.16 66 640: 100%|██████████| 47429/47429 [2:18:29<00:00, 5.71it/s] Class Images Instances Box(P R mAP50 mAP50-95): 71%|███████ | 2105/2968 [06:00<06:03, 2.37it/s]WARNING ⚠️ NMS time limit 2.100s exceeded Class Images Instances Box(P R mAP50 mAP50-95): 71%|███████ | 2108/2968 [27:17<72:53:17, 305.11s/it]WARNING ⚠️ NMS time limit 2.100s exceeded Class Images Instances Box(P R mAP50 mAP50-95): 71%|███████ | 2109/2968 [3:38:58<871:14:46, 3651.32s/it]^C

Environment

yolov8,cuda gpu

Minimal Reproducible Example

No response

Additional

please help me resolve this

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

github-actions[bot] commented 11 months ago

👋 Hello @sanjanak-rbg, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.8.0 with all requirements.txt installed including PyTorch>=1.8. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

Notebooks with free GPU:
Google Cloud Deep Learning VM. See GCP Quickstart Guide
Amazon Deep Learning AMI. See AWS Quickstart Guide
Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

glenn-jocher commented 11 months ago

@sanjanak-rbg hi there! Thank you for reaching out.

The warning message you encountered, "WARNING ⚠️ NMS time limit 2.100s exceeded," indicates that the Non-Maximum Suppression (NMS) algorithm is taking longer than the specified time limit to execute. This can occur when the number of bounding box predictions is very high or the NMS algorithm implementation is slow.

To address this, you have a few options:

Use a smaller image size during validation. By reducing the image size, you can decrease the number of bounding box predictions and potentially reduce the execution time of the NMS algorithm.
Adjust the NMS time limit. You can increase the time limit for NMS by modifying the code.
Improve the efficiency of the NMS algorithm implementation. If you are comfortable with modifying the YOLOv5 code, you can explore optimizing the NMS algorithm for your specific hardware or use a more efficient NMS algorithm implementation.

Please note that these options might have trade-offs in terms of accuracy or execution time. Additionally, sharing more details, such as your dataset characteristics and hardware configuration, can help the community provide more specific suggestions.

Thanks for your contribution and for being part of the YOLOv5 community! Let us know if you have any further questions or need additional assistance.

github-actions[bot] commented 10 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

twilson1111 commented 6 months ago

I am seeing the same issue with my M2 macbook air. Additionally, it coincides with my 'mps' training not converging (ie: loss not decreasing), whereas 'cpu' training does converge.

glenn-jocher commented 6 months ago

@twilson1111 hey there! 😊 It sounds like you’re experiencing a couple of challenges with training on your M2 MacBook Air.

For the NMS time limit warning and the non-converging issue on 'mps' (Metal Performance Shaders), this is likely related to the differences in computing capabilities and optimizations between 'mps' and 'cpu'. Metal tends to work differently with parallel computations compared to CPU, which might explain the convergence issue.

A quick suggestion for the convergence problem: try adjusting your learning rate or batch size when training with 'mps'. Sometimes, slight tweaks here can significantly impact training dynamics.

For the NMS time limit exceeded warning, it's generally a performance warning rather than an error that would stop your training. If it doesn't significantly impact your training time or outcomes, it might be okay to proceed as is.

Remember, training on GPU/Metal often requires different hyperparameters tuning compared to CPU to achieve optimal results.

Let me know if these suggestions help or if you have more queries!

twilson1111 commented 6 months ago

@twilson1111 hey there! 😊 It sounds like you’re experiencing a couple of challenges with training on your M2 MacBook Air.

For the NMS time limit warning and the non-converging issue on 'mps' (Metal Performance Shaders), this is likely related to the differences in computing capabilities and optimizations between 'mps' and 'cpu'. Metal tends to work differently with parallel computations compared to CPU, which might explain the convergence issue.

A quick suggestion for the convergence problem: try adjusting your learning rate or batch size when training with 'mps'. Sometimes, slight tweaks here can significantly impact training dynamics.

For the NMS time limit exceeded warning, it's generally a performance warning rather than an error that would stop your training. If it doesn't significantly impact your training time or outcomes, it might be okay to proceed as is.

Remember, training on GPU/Metal often requires different hyperparameters tuning compared to CPU to achieve optimal results.

Let me know if these suggestions help or if you have more queries!

Hi Glenn,

Thank you for your assistance. I have tried different batch sizes but haven't had much luck - although with my hardware (a new Macbook air) I cannot go past 16 without everything getting too slow. With regards to learning rates, would it be possible for you and Ultralytics to come up with a suggestion for which learning rate/optimizer to use, or in which direction? The reason I ask is, the optimizer for my dataset is automatic which is overriding any learning rate change I make, and there are a bunch of optimizers so if I go manual, it's hard to know where to start.
Thank you for trying to help. I do think it's quite unfortunate that 'mps' just doesn't converge, given that I have a newer model M GPU, and that we don't have much to point towards other than nebulous 'accuracy' concerns. I wonder if you could help point us unfortunate mac users in a more concrete direction for getting the ultralytics library to work? I know that the various 'mps' Ultralytics issues documented online have been going on for > 1 year, so it's concerning regarding whether this is actually a supported platform or on its way out. There seem to be several areas where the Apple ecosystem is becoming less compatible with cross-platform AI libraries.

Update - I have gotten some results from changing optimizer to "Adam" and reducing the Lr0 by a factor of 4x from where the auto-optimizer had it (0.00025). This allows the system to move in the right direction using 'mps' in a noticable way over the first 10 epochs.

glenn-jocher commented 6 months ago

Hi @twilson1111,

Great to hear you've made progress with Adam optimizer and adjusting the learning rate! 👍 It's a step in the right direction. The challenges with 'mps' on Mac hardware, especially with convergence, are indeed tricky. The specific nuances of Metal's compute model, combined with YOLOv5's architecture and your unique dataset, can make finding the perfect training configuration a bit of an art form.

Given your situation, starting with Adam was a good choice, as it's known for being well-suited to a wide range of problems. The learning rate adjustment sounds like it was effective, too. For further tuning, consider experimenting with the weight decay and the learning rate scheduler. These can also significantly impact model convergence and performance.

It's worth noting that YOLOv5 and other AI libraries do face challenges in ensuring seamless compatibility across the rapidly evolving hardware landscape, including Apple's M-series chips. We're always working to improve support and optimize performance across all platforms, including 'mps'. Your feedback and contributions from the community are invaluable as we navigate these complexities together.

Remember, experimentation is key. Small adjustments can sometimes yield surprising improvements. If you hit another roadblock, feel free to share your findings and questions. We're here to help!

Best of luck with your continued training, and thanks for being a proactive part of the YOLOv5 community!

ErfanAlaeii commented 4 months ago

Code Optimization: Review the relevant code and improve the parts that can be optimized. Using more optimal algorithms and reducing time complexity can be useful.

Increasing system resources: Increasing computing resources such as CPU and RAM can help improve the speed of operations.

glenn-jocher commented 4 months ago

Hi there!

For code optimization in YOLOv5, focusing on algorithm efficiency and reducing time complexity is indeed a great approach. You might consider profiling your code to identify bottlenecks and then optimize those specific areas. For example, optimizing loop structures or leveraging more efficient data handling can yield significant improvements.

Regarding system resources, increasing CPU and RAM can definitely help, especially when dealing with large datasets or when you're aiming to decrease training times. If feasible, upgrading your hardware or using a more powerful server could provide the boost you need.

Keep experimenting and profiling to find the best balance for your specific use case! 😊 If you have specific sections of code you're concerned about, feel free to share, and we can look into those together.

eripe970 commented 4 months ago

Adjust the NMS time limit. You can increase the time limit for NMS by modifying the code.

Maybe this could be a parameter to the training command?

glenn-jocher commented 4 months ago

Hi @eripe970,

That's a great suggestion! 🎉 Making the NMS time limit configurable via a parameter in the training command could provide more flexibility for users. We'll consider this for future updates. In the meantime, you can manually adjust the NMS time limit in the code to better suit your needs.

Thanks for your input and for being an active part of the YOLOv5 community! 😊

ultralytics / yolov5