True Positive, False Positive ,False Negative, True Negative

[Shanky71]

Search before asking

• [X] I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Hi @glenn-jocher What does True Positive, False Positive,False negative and True Negative means in yolov5. I know it's a very basic question but several references are giving me several answers. Can you explain me with some example?

Additional

No response

[glenn-jocher]

@Shanky71 true Positive (TP): Correctly predicted object by YOLOv5 💯 False Positive (FP): Incorrectly predicted object by YOLOv5 🚫 False Negative (FN): Missed detection by YOLOv5 ❌ True Negative (TN): Not applicable to object detection 🤷‍♂️

For more details, check out the YOLOv5 docs at https://docs.ultralytics.com/yolov5/ 📚

[Shanky71]

@glenn-jocher what about the misclassification and wrong bounding box? Under which category will it lie. I'm a bit confused about that

[glenn-jocher]

@Shanky71 Misclassification and wrong bounding boxes typically fall under false positives, as they represent incorrect predictions made by the model. For more details, you can refer to the YOLOv5 documentation at https://docs.ultralytics.com/yolov5/ 📚

[Shanky71]

@glenn-jocher I am confused how you are defining the wrong bounding boxes as the yolov5 with generate multiple bounding box. If True positive: 88.6% False positive: 9.6% False negative: 11.4% how can I calculate the % for wrong bounding box and misclassification separately. I checked manually about the % of misclassfication of the validation dataset , it came out as 0.64%, so does it mean I have 8.96% of wrong bounding boxes? To confirm this I used IOU overlapping between annotations of my validation dataset and generating the detections from yolov5, the mean IOU value came out to be 0.995.

Why so much difference ?

[glenn-jocher]

@Shanky71 During evaluation, misclassification and wrong bounding boxes can be part of false positives. The difference in percentages can arise from the nuanced definitions and evaluations of misclassification, wrong bounding boxes, and false positives. The high IOU value of 0.995 indicates an accurate localization between annotations and YOLOv5 detections, emphasizing the model's ability to generate precise bounding boxes.

Keep in mind that percentages can vary based on specific use cases, dataset characteristics, and evaluation methodologies. For in-depth analysis, it's recommended to refer to the YOLOv5 documentation, and consider engaging the YOLO community for diverse perspectives.

If you have further questions, the Ultralytics team and the YOLO community are here to help!

[Shanky71]

@glenn-jocher Thank you for your kind help. So does it mean there is no way to find the % for wrong bounding box and misclassification separately?

[glenn-jocher]

@Shanky71 That's correct! In object detection, the traditionally used metrics like True Positive, False Positive, False Negative, and True Negative do not directly provide separate percentages for wrong bounding boxes and misclassifications. These nuances are typically subsumed within the false positive category. To understand these aspects separately, custom evaluation methodologies and metrics may need to be developed based on specific use cases and requirements.

For further details and insights, feel free to explore the YOLOv5 documentation at https://docs.ultralytics.com/yolov5/ and engage with the YOLO community. We're here to support you!

[Shanky71]

@glenn-jocher Thank you so much for clearing my doubts. Thanks for your kind help and time.

[glenn-jocher]

You're welcome, @Shanky71! I'm glad I could help. If you have any more questions in the future, don't hesitate to ask. The YOLOv5 community and the Ultralytics team are here to support you. Good luck with your projects! 😊👍

[github-actions[bot]]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[Shanky71]

@glenn-jocher Hey I have a small doubt

1. A detection predicted by the model although there is no ground truth object present there, under which category will it come? True positive, False Positive, False Negative or True Negative
2. While training the dataset why after each epoch we are getting the precision and recall for validation data, why not after training the whole training data for 300 epochs and then validation for only one time instead of 300 times?
3. Why is metrics/precision and metrics/recall. I am interested in plotting only Precision-Recall curve. When I tried plotting metrics/precision vs metrics/recall , I am getting a very strange curve.

[glenn-jocher]

Hello @Shanky71! Let's address your questions one by one:

1. Detection with no ground truth object: This scenario is classified as a False Positive (FP). It indicates that the model predicted an object where there was none, which is a type of error.

2. Validation after each epoch: The reason for evaluating precision and recall on the validation dataset after each epoch, rather than only after all epochs, is to monitor the model's performance and generalization ability over time. This frequent evaluation helps in identifying overfitting, underfitting, and the best epoch to stop training to avoid overfitting. It also allows for early stopping if the model's performance on the validation set starts to degrade, saving time and resources.

3. Precision-Recall curve issue: The Precision-Recall (PR) curve you're trying to plot should ideally show the trade-off between precision and recall for different thresholds. If the curve looks strange, it might be due to incorrect plotting or data handling. Ensure you're correctly extracting precision and recall values at various thresholds. Also, consider using established libraries or tools that automatically calculate and plot PR curves to avoid such issues.

Remember, the YOLOv5 community and the Ultralytics team are always here to help. If you have further questions or need clarification, feel free to reach out!

[Shanky71]

@glenn-jocher Thanks Glenn for the help. Just a quick clarification, is it possible to draw the PR curve for all classes instead of individual class separately without training the data again. Also you have given a results.csv can it be used for it? If yes can you help me out through this?

Also can you help me in understanding the importance of mAP@0.5 and mAP@0.5:0.95? Which one is more useful and how can we understand them so that we can differentiate between them.

[glenn-jocher]

Certainly, @Shanky71! Let's dive into your queries:

1. Drawing the PR curve for all classes: It is indeed possible to draw a Precision-Recall (PR) curve that aggregates the performance across all classes without retraining your model. This would involve averaging precision and recall values across all classes at various thresholds. However, this aggregation might obscure class-specific performance details. The results.csv file generated by YOLOv5 during training or evaluation contains detailed metrics for each class and can be used to plot these curves. You would need to extract precision and recall values for each class and then compute their average or use a library that supports multi-class PR curve plotting.

2. Understanding mAP@0.5 and mAP@0.5:0.95:

   • mAP@0.5: This metric calculates the mean Average Precision (mAP) considering a detection correct if the Intersection over Union (IoU) with a ground truth object is greater than 0.5. It focuses on whether the model can locate objects reasonably well, without being too strict about the exactness of the bounding box.
   • mAP@0.5:0.95: This metric averages the mAP calculated at different IoU thresholds, from 0.5 to 0.95 (in steps of 0.05). It provides a more comprehensive assessment of the model's performance across a range of criteria from loose to strict. This metric is more challenging and gives a better overall picture of how well the model is performing in terms of both localization and detection.

Which one is more useful? It depends on your application:

• If precise localization of objects is crucial (e.g., for robotic manipulation), mAP@0.5:0.95 gives a more balanced view of the model's performance.
• If the focus is more on detection rather than precise localization, mAP@0.5 might be sufficient.

Both metrics are important and provide valuable insights into different aspects of the model's performance. For a comprehensive evaluation, it's advisable to consider both.

If you have more questions or need further assistance, feel free to reach out. The YOLOv5 community and the Ultralytics team are here to support you!

[Shanky71]

@glenn-jocher Hey I have a small doubt

1. A detection predicted by the model although there is no ground truth object present there, under which category will it come? True positive, False Positive, False Negative or True Negative

2. While training the dataset why after each epoch we are getting the precision and recall for validation data, why not after training the whole training data for 300 epochs and then validation for only one time instead of 300 times?

3. Why is metrics/precision and metrics/recall. I am interested in plotting only Precision-Recall curve. When I tried plotting metrics/precision vs metrics/recall , I am getting a very strange curve.
   ![PR curve](https://private-user-images.githubusercontent.com/72705901/308531334-b425f844-4c43-4606-9cf8-7ec7aafb8979.jpg?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MDk1MjY3ODMsIm5iZiI6MTcwOTUyNjQ4MywicGF0aCI6Ii83MjcwNTkwMS8zMDg1MzEzMzQtYjQyNWY4NDQtNGM0My00NjA2LTljZjgtN2VjN2FhZmI4OTc5LmpwZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDAzMDQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwMzA0VDA0MjgwM1omWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTY4MGExNGM3NjdkZDBlMzYzMTRjNWExNWNhNjc3MzA4NTA4ZjJhZDczOGI1OWU3NTU1MzBhZTVhOWEzMWQwN2MmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0._xisQerN25qWrzPIwQidjJ6-Dy3A2nUeXxMrGeVwg04)

Hey @glenn-jocher, In the second question, what changes can occur in the model if I don't want to run the model on validation dataset and just want to do the validation after the training of the model is over. Won't it save more time and computational resources? Also then I feel validation should serve the purpose for the test dataset too

[glenn-jocher]

Hello @Shanky71,

Skipping validation during training and only validating after training completes would indeed save time and computational resources during the training process. However, there are several reasons why performing validation at regular intervals during training is beneficial and often crucial:

1. Model Performance Monitoring: Regular validation allows you to monitor the model's performance on unseen data throughout the training process. This helps in identifying if and when the model starts overfitting or underfitting.

2. Early Stopping: By evaluating the model on a validation set at regular intervals, you can implement early stopping to terminate training if the model's performance on the validation set begins to degrade or fails to improve. This can save resources by avoiding unnecessary training epochs.

3. Model Selection: Validating after each epoch helps in selecting the best model based on its performance on the validation set. Without regular validation, you would only have the final model to evaluate, which might not be the best version of the model during the training process.

4. Hyperparameter Tuning: Regular validation is crucial for hyperparameter tuning. It allows you to adjust learning rates, regularization parameters, and other hyperparameters based on the model's performance on the validation set.

Skipping validation and using the validation set as a test set at the end of training would not provide these benefits. The test set should ideally be a completely separate dataset that the model has never seen during training or hyperparameter tuning. Its purpose is to evaluate the final model's performance and generalize ability to new, unseen data.

If computational resources are a concern, you might consider validating less frequently (e.g., every few epochs instead of after every single epoch) or using a smaller subset of the validation data for quicker evaluations. However, completely skipping validation until the end of training is generally not recommended for the reasons mentioned above.

I hope this clarifies your doubts! If you have further questions, feel free to ask.

[Shanky71]

Yeah it clears my doubt , thanks for the quick help @glenn-jocher . Performance metrics (Precision, Recall etc.) should be considered after training/validation/test? Which one will describe my model well?

[glenn-jocher]

You're welcome, @Shanky71! I'm glad to hear that your doubts are cleared.

Regarding performance metrics like Precision, Recall, and others, here's how and when they should be considered:

1. After Training (Validation Metrics): During and after training, you'll use these metrics on the validation dataset to tune your model, select the best model, and adjust hyperparameters. Metrics on the validation set give you an idea of how well your model generalizes to unseen data. However, since the model's hyperparameters might be tuned based on the performance on the validation set, these metrics might not fully represent the model's ability to generalize to completely new data.

2. After Model Selection (Test Metrics): The most accurate assessment of your model's performance comes from evaluating it on the test set. The test set should be a completely separate dataset that has not been used during training or validation. Metrics calculated on the test set are crucial because they provide an unbiased evaluation of the model's generalization capability to new, unseen data.

Which one will describe my model well? Metrics from both validation and test phases are important, but for different reasons:

• Validation metrics are essential for model development, including training, tuning, and selection.
• Test metrics are the ultimate measure of your model's performance and generalization ability. They tell you how well your model is expected to perform in the real world on data it has never seen before.

Always ensure that your test dataset is well-curated and representative of the real-world scenarios where the model will be deployed. This ensures that the test metrics accurately reflect the model's effectiveness and usability in practical applications.

If you have any more questions or need further assistance, feel free to reach out. Happy to help!

[Shanky71]

Hey @glenn-jocher Thanks for your valuable time and help. Is it possible that if I have divided my data into training and testing without validation set, should I run the testing set only once?

[glenn-jocher]

You're welcome, @Shanky71! I'm here to help.

If you've divided your data into only training and testing sets, without a separate validation set, it's crucial to maintain the integrity of the testing process to ensure that the evaluation of your model is unbiased and reflects its true performance. Here are some guidelines:

1. Run the Testing Set Only Once: Ideally, you should run your model on the testing set only once, after all model training, tuning, and selection have been completed. This approach ensures that the test metrics you obtain are a genuine measure of how well your model can generalize to new, unseen data.

2. Why Only Once?: The reason for limiting the evaluation on the test set to a single run is to prevent any inadvertent "leakage" of test data information into the model training process. If you were to use the test set multiple times, especially if you make changes to your model or training process based on test set performance, you risk overfitting to the test set. This would make your model's performance on the test set an unreliable indicator of its performance on truly unseen data.

3. No Validation Set?: If you haven't set aside a validation set, consider using techniques like cross-validation during the training phase to help with model selection and hyperparameter tuning without needing a separate validation set. Cross-validation involves dividing your training data into several subsets, training your model multiple times with different subsets held out as a "pseudo-validation" set, and averaging the results. This can help simulate the benefits of having a separate validation set.

Remember, the goal of the test set is to simulate how well your model performs on data it has never encountered, mimicking real-world conditions as closely as possible. Keeping the test set strictly for final evaluation helps maintain the integrity of this simulation.

If you have any more questions or need further clarification, feel free to ask. Good luck with your model development!

[Shanky71]

Hey @glenn-jocher python detect.py --img-size 2048 --data data/custom.yaml --weights runs/train/exp25/weights/best.pt --source test_image --save-txt I am not able to find test.py for yolov5. So is it okay to use detect.py here for testing?

How should I modify it such that it such that only training can be done without validation and I can just test it once. Also I am interested in obtaining the metrics like precision , recall, map@0.5 etc for the test images

[glenn-jocher]

Hello @Shanky71,

Yes, you can use detect.py for testing your model on a set of images if you're looking to perform inference and see the model's predictions. However, detect.py is primarily designed for inference and might not directly provide detailed metrics like precision, recall, mAP@0.5, etc., for your test images in the same way that a dedicated evaluation script would.

To obtain detailed evaluation metrics on a test set without using a validation set during training, you can follow these steps:

1. Training Without Validation: Modify your training command to skip the validation phase. This can be done by setting the --noval flag during training if you're using a version of YOLOv5 that supports it. If not, you might need to ensure your data/custom.yaml does not specify a validation dataset. However, be aware that skipping validation means you won't be able to monitor the model's performance on unseen data during training, which is generally not recommended.

2. Testing for Metrics: To evaluate your model on a test set and obtain metrics like precision, recall, and mAP, you would typically use a script designed for evaluation, such as val.py in YOLOv5. This script can evaluate your model on a specified dataset and compute these metrics.

   You can run something like this:

   python val.py --weights runs/train/exp25/weights/best.pt --data data/custom.yaml --img 2048 --task test --verbose

   Make sure your data/custom.yaml file includes a path to your test set under the test: key. The --task test option tells the script to evaluate the model on the test set specified in your dataset YAML file. The --verbose flag will provide detailed output.

   This approach assumes that your test dataset is properly labeled, as the script needs ground truth annotations to compute metrics like precision and recall.

Remember, the distinction between validation and test datasets is in how they're used during the model development process. The validation set is used to tune the model and make decisions about hyperparameters, while the test set is used to evaluate the model's final performance. If you skip validation, be cautious not to inadvertently tune your model based on test set performance, as this would bias your evaluation.

If you have further questions or need more assistance, feel free to reach out. Good luck with your testing!

[Shanky71]

@glenn-jocher Thank you so much for helping me throughout the process. I really appreciate that. Just a small doubt here again. After doing all of this I tried running that, it was running on my CPU instead of my GPU. How can I make it run on GPU instead of CPU. I already used --device 0 in the code snippet still it's running on CPU? Any comments why it's happening?

import utils display = utils.notebook_init()

it shows me CPU instead of GPU. How to change that

YOLOv5 2024-3-6 Python-3.11.5 torch-2.1.2 CPU Setup complete (96 CPUs, 382.7 GB RAM, 717.4/930.8 GB disk)

[glenn-jocher]

I'm glad to have been able to help you, @Shanky71. Regarding your issue with the model running on the CPU instead of the GPU, let's address that.

The fact that YOLOv5 is defaulting to CPU despite specifying --device 0 suggests a few potential issues to check:

1. PyTorch GPU Support: Ensure that your PyTorch installation supports CUDA. You can verify this by running:

   import torch
   print(torch.cuda.is_available())

   If this prints False, PyTorch is not detecting any CUDA-capable devices, which means it can't utilize your GPU.

2. CUDA Installation: Make sure you have CUDA installed and it's a version compatible with your PyTorch installation. You can check the compatible versions on the PyTorch website.

3. NVIDIA Drivers: Ensure your NVIDIA drivers are up to date. Outdated drivers can cause issues with CUDA and, by extension, PyTorch's ability to utilize the GPU.

4. Environment Variables: Sometimes, setting the CUDA_VISIBLE_DEVICES environment variable explicitly to the GPU you want to use (e.g., export CUDA_VISIBLE_DEVICES=0) before running your script can help, especially if you have multiple GPUs.

5. PyTorch Installation: If you've verified that CUDA is installed correctly and your NVIDIA drivers are up to date, but PyTorch still doesn't recognize your GPU, you might need to reinstall PyTorch. Ensure you select the correct version for your CUDA version from the PyTorch installation command generator on their website.

6. YOLOv5 Dependency: Ensure all dependencies for YOLOv5 are correctly installed, especially those related to CUDA and GPU support. Sometimes, a missing or incompatible dependency can cause fallback to CPU.

If after checking these points you're still facing issues, it might be helpful to create a new environment and reinstall the necessary packages, ensuring compatibility between PyTorch, CUDA, and your system's drivers.

Remember, successfully running on GPU significantly speeds up both training and inference processes, so it's worth ensuring everything is correctly set up. If you continue to encounter difficulties, consider seeking help with specific error messages or system configurations, as the issue might be more nuanced.

I hope this helps you resolve the issue! If you have any more questions or need further assistance, feel free to ask.

[Shanky71]

Hey @glenn-jocher , the training started but it ended abruptly. I am still not able to figure how to rectify it

Starting training for 1 epochs...

  Epoch    GPU_mem   box_loss   obj_loss   cls_loss  Instances       Size
    0/0      65.2G     0.1607     0.2422     0.0445       5201       2048: 100%|##########| 40/40 [09:40<00:00, 14.52s/it]
             Class     Images  Instances          P          R      mAP50   mAP50-95:   0%|          | 0/4 [00:01<?, ?it/s]

Traceback (most recent call last): File "C:\Users\user\yolov5\train.py", line 848, in main(opt) File "C:\Users\user\yolov5\train.py", line 623, in main train(opt.hyp, opt, device, callbacks) File "C:\Users\user\yolov5\train.py", line 426, in train results, maps, _ = validate.run( ^^^^^^^^^^^^^ File "C:\Users\user\yolov5_training_env\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\yolov5\val.py", line 246, in run preds = non_max_suppression( ^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\yolov5\utils\general.py", line 1099, in non_max_suppression i = torchvision.ops.nms(boxes, scores, iou_thres) # NMS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\yolov5_training_env\Lib\site-packages\torchvision\ops\boxes.py", line 41, in nms return torch.ops.torchvision.nms(boxes, scores, iou_threshold) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\user\yolov5_training_env\Lib\site-packages\torch_ops.py", line 755, in call return self._op(args, **(kwargs or {})) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ NotImplementedError: Could not run 'torchvision::nms' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'torchvision::nms' is only available for these backends: [CPU, Meta, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradMPS, AutogradXPU, AutogradHPU, AutogradLazy, AutogradMeta, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

CPU: registered at C:\actions-runner_work\vision\vision\pytorch\vision\torchvision\csrc\ops\cpu\nms_kernel.cpp:112 [kernel] Meta: registered at /dev/null:440 [kernel] QuantizedCPU: registered at C:\actions-runner_work\vision\vision\pytorch\vision\torchvision\csrc\ops\quantized\cpu\qnms_kernel.cpp:124 [kernel] BackendSelect: fallthrough registered at ..\aten\src\ATen\core\BackendSelectFallbackKernel.cpp:3 [backend fallback] Python: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:154 [backend fallback] FuncTorchDynamicLayerBackMode: registered at ..\aten\src\ATen\functorch\DynamicLayer.cpp:498 [backend fallback] Functionalize: registered at ..\aten\src\ATen\FunctionalizeFallbackKernel.cpp:324 [backend fallback] Named: registered at ..\aten\src\ATen\core\NamedRegistrations.cpp:7 [backend fallback] Conjugate: registered at ..\aten\src\ATen\ConjugateFallback.cpp:17 [backend fallback] Negative: registered at ..\aten\src\ATen\native\NegateFallback.cpp:19 [backend fallback] ZeroTensor: registered at ..\aten\src\ATen\ZeroTensorFallback.cpp:86 [backend fallback] ADInplaceOrView: fallthrough registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:86 [backend fallback] AutogradOther: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:53 [backend fallback] AutogradCPU: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:57 [backend fallback] AutogradCUDA: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:65 [backend fallback] AutogradXLA: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:69 [backend fallback] AutogradMPS: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:77 [backend fallback] AutogradXPU: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:61 [backend fallback] AutogradHPU: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:90 [backend fallback] AutogradLazy: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:73 [backend fallback] AutogradMeta: registered at ..\aten\src\ATen\core\VariableFallbackKernel.cpp:81 [backend fallback] Tracer: registered at ..\torch\csrc\autograd\TraceTypeManual.cpp:297 [backend fallback] AutocastCPU: registered at C:\actions-runner_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autocast\nms_kernel.cpp:34 [kernel] AutocastCUDA: registered at C:\actions-runner_work\vision\vision\pytorch\vision\torchvision\csrc\ops\autocast\nms_kernel.cpp:27 [kernel] FuncTorchBatched: registered at ..\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:720 [backend fallback] BatchedNestedTensor: registered at ..\aten\src\ATen\functorch\LegacyBatchingRegistrations.cpp:746 [backend fallback] FuncTorchVmapMode: fallthrough registered at ..\aten\src\ATen\functorch\VmapModeRegistrations.cpp:28 [backend fallback] Batched: registered at ..\aten\src\ATen\LegacyBatchingRegistrations.cpp:1075 [backend fallback] VmapMode: fallthrough registered at ..\aten\src\ATen\VmapModeRegistrations.cpp:33 [backend fallback] FuncTorchGradWrapper: registered at ..\aten\src\ATen\functorch\TensorWrapper.cpp:203 [backend fallback] PythonTLSSnapshot: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:162 [backend fallback] FuncTorchDynamicLayerFrontMode: registered at ..\aten\src\ATen\functorch\DynamicLayer.cpp:494 [backend fallback] PreDispatch: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:166 [backend fallback] PythonDispatcher: registered at ..\aten\src\ATen\core\PythonFallbackKernel.cpp:158 [backend fallback]

[glenn-jocher]

@Shanky71 it looks like you're encountering a NotImplementedError related to the torchvision::nms operation, which is a non-maximum suppression function crucial for object detection tasks. This error typically indicates that PyTorch and torchvision are attempting to use an operation on the GPU that isn't supported by the installed versions of PyTorch or torchvision for the CUDA backend you're using.

Here are a few steps to troubleshoot and potentially resolve this issue:

1. Compatibility Check: Ensure that your PyTorch and torchvision versions are compatible with each other and with your CUDA version. PyTorch's and torchvision's documentation or their forums can provide information on compatible versions.

2. Update PyTorch and torchvision: If there's a mismatch or if you're not using the latest versions, consider updating both PyTorch and torchvision. Use the commands below, but make sure to choose the versions that match your CUDA version:

   pip install torch --upgrade
   pip install torchvision --upgrade

   Visit the official PyTorch website to get the correct installation command that matches your environment.

3. CUDA Version: Double-check your CUDA version and ensure it's supported by the versions of PyTorch and torchvision you're using. You can check your CUDA version with nvcc --version or nvidia-smi.

4. Environment Variables: Sometimes, setting the right environment variables can help, especially if you have multiple CUDA versions installed. Make sure your PATH and LD_LIBRARY_PATH (Linux) or CUDA_PATH (Windows) point to the correct CUDA installation directories.

5. Reinstallation: If the above steps don't resolve the issue, consider creating a new virtual environment and reinstalling PyTorch, torchvision, and CUDA Toolkit from scratch, ensuring compatibility between them.

6. PyTorch Forums: If you continue to face issues, the PyTorch forums are an excellent resource for help. There might be others who have faced similar issues, or you can post your problem there for community assistance.

Remember, the key to resolving such issues often lies in ensuring compatibility between PyTorch, torchvision, and CUDA. Keeping everything up-to-date and aligned with the versions supported by your hardware usually helps avoid such errors.

If you need further assistance, feel free to ask. Good luck!

[Shanky71]

Thank you so much @glenn-jocher . I was able to figure out the issue with your kind help. Thanks for helping throughout.

[glenn-jocher]

You're very welcome, @Shanky71! I'm thrilled to hear that you were able to resolve the issue. Remember, the journey of learning and working with technologies like YOLOv5 is a collaborative effort, and I'm here to support you along the way. If you have any more questions or need further assistance in the future, don't hesitate to reach out. Best of luck with your projects, and keep up the great work!

[Shanky71]

Hey @glenn-jocher Can you please tell me what happens in each epoch with the validation and training dataset? What is the input and output we get after each epoch

[glenn-jocher]

Absolutely, @Shanky71! Let's break down what happens during each epoch in the training process, especially focusing on how training and validation datasets are utilized. This explanation is general and applies broadly to neural network training, including models like YOLOv5.

Epoch Overview

An epoch represents one complete pass through the entire training dataset. During an epoch, the model will see every example in the training set once. The goal of each epoch is to update the model's weights to minimize the loss function, thereby improving the model's predictions.

Training Phase

Input: The input during the training phase of an epoch is the training dataset, which consists of labeled data points. In the context of YOLOv5 and object detection, these data points are images with corresponding bounding boxes and class labels for objects present in the images.

Process:

1. Forward Pass: For each batch of data from the training set, the model performs a forward pass, making predictions based on its current weights.
2. Calculate Loss: The predictions are compared to the actual labels using a loss function, which quantifies how far off the predictions are from the truth.
3. Backward Pass: Through backpropagation, gradients of this loss are calculated with respect to the model's weights.
4. Update Weights: The model's weights are updated using an optimization algorithm (like SGD or Adam) in an attempt to reduce the loss.

Output: After each epoch, you typically get the average training loss, which indicates how well the model is fitting the training data. You might also track other metrics like accuracy, depending on your specific task.

Validation Phase

Input: The input during the validation phase is the validation dataset. This dataset is separate from the training dataset and is not used to update the model's weights. Its purpose is to provide an unbiased evaluation of a model fit on the training dataset.

Process:

1. Forward Pass Only: The model performs a forward pass on the validation dataset, making predictions based on its current (post-training epoch) weights.
2. Calculate Metrics: Since the model's weights are not updated during this phase, the focus is on calculating performance metrics (e.g., validation loss, precision, recall, mAP) to evaluate how well the model generalizes to unseen data.

Output: After the validation phase of an epoch, you receive metrics that indicate the model's performance on the validation set. These metrics are crucial for understanding the model's generalization ability and for tuning hyperparameters. Unlike the training phase, there's no weight update here.

Importance of Training and Validation Phases

• Training Phase: Aims to fit the model to the training data as closely as possible, within the constraints of the model architecture and loss function.
• Validation Phase: Provides a check against overfitting by using unseen data to evaluate the model's performance. It helps in tuning the model and deciding when to stop training.

After each epoch, by comparing the training and validation metrics, you can get insights into how well your model is learning and generalizing. Ideally, both training and validation losses should decrease over time. If the validation loss starts to increase, it might be a sign of overfitting to the training data.

I hope this clarifies the process for you! If you have any more questions, feel free to ask.

[Shanky71]

Thank you so much @glenn-jocher .

[glenn-jocher]

@Shanky71 you're welcome! I'm glad I could help. If you have any more questions in the future or need further assistance, don't hesitate to reach out. Happy coding, and best of luck with your projects! 😊

[Shanky71]

Hey @glenn-jocher Can you please answer this in context of yolov5. What are hyperparameters? How validation helps in hyperparameters tuning? How they help in preventing overfitting and underfitting?

[glenn-jocher]

Hey @Shanky71! Great questions. In the context of YOLOv5:

Hyperparameters are the configuration settings used to structure the training process. Examples include learning rate, batch size, and epochs. Unlike model parameters, hyperparameters are not learned from the data but set before training begins.

Validation and Hyperparameter Tuning: The validation set helps in hyperparameter tuning by providing feedback on model performance on unseen data. Adjusting hyperparameters based on validation metrics can improve model generalization, preventing overfitting (model too complex, fitting training data too closely) and underfitting (model too simple, inadequate learning from training data).

For example, if validation loss is higher than training loss, it might indicate overfitting. You might lower learning rate or increase regularization. If both losses are high, it could be underfitting, where you might increase model complexity or learning rate.

In short, validation plays a critical role in selecting hyperparameters that balance the trade-off between model complexity and its ability to generalize well to unseen data.

Hope this helps! 😊

[Shanky71]

https://github.com/ultralytics/yolov5/issues/12555#issuecomment-1968906773

Hey @glenn-jocher What inference can we draw from the PR curve?

[glenn-jocher]

@Shanky71 hey there! 😊 The Precision-Recall (PR) curve is a fantastic tool for evaluating the performance of your object detection model, especially in contexts where classes are imbalanced. Precision tells us how many of the predicted positive cases were actually positive, while Recall (or sensitivity) measures how many of the actual positive cases were correctly identified by the model.

From the PR curve, you can infer:

• The balance between precision and recall: Ideally, you want both high precision and high recall, but usually improving one can reduce the other.
• Model performance across different thresholds: Adjusting the classification threshold can shift the balance between precision and recall, which is directly visible on the PR curve.
• Area Under the Curve (AUC): A higher AUC indicates a better model performance, integrating across all possible classification thresholds.

In summary, the PR curve provides insights into how well your model distinguishes between classes, especially in scenarios where one class dominance might skew traditional accuracy metrics. Hope this clarifies your query!

[Shanky71]

Thank you so much @glenn-jocher for helping me in understanding the basic doubts about the model. I really appreciate your kind help.

[glenn-jocher]

You're very welcome, @Shanky71! 😊 I'm glad I could assist. If you have any more questions or need further help down the line, feel free to reach out. Happy coding, and best of luck with your projects!

[Shanky71]

Hey @glenn-jocher does precision depends on recall? the columns “precision” (for which recall?) and “recall” (for which precision?) Is there any missing information

<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">

Vehicle Class | Precision | Recall | mAP@0.5 | mAP@0.5:0.95 -- | -- | -- | -- | -- All | 0.90 | 0.88 | 0.91 | 0.59 Class1 | 0.90 | 0.88 | 0.93 | 0.54 Class2 | 0.96 | 0.95 | 0.97 | 0.75 Class3 | 0.97 | 0.89 | 0.91 | 0.58 Class4 | 0.87 | 0.77 | 0.89 | 0.69 Class5 | 0.90 | 0.88 | 0.90 | 0.65

Furthermore, if mAP, i.e., area under the curve depends on IoU, there must be another control varying P and R (otherwise, you would not have a precision-recall curve for a fixed IoU from which to calculate mAP@0.95]

[glenn-jocher]

Hey there! It seems there's a bit of confusion about how precision, recall, and mAP relate to each other, especially in the context of YOLOv5. Let's clarify:

Precision and Recall Relationship: Precision and recall are inversely related in many scenarios. Improving precision often reduces recall, and vice versa. This is because precision focuses on the quality of positive predictions, while recall focuses on identifying all actual positives, quality aside. They don't depend on each other per se but are affected by the decision threshold: lowering it increases recall and potentially lowers precision, while raising it does the opposite.

"For which" Precision and Recall? When we talk about precision and recall without specifying, we're generally referring to these metrics computed over a range of decision thresholds, which gives us an aggregate understanding of model performance across different levels of certainty for its predictions.

mAP and IoU: mAP (mean Average Precision) considers the entire precision-recall curve across different thresholds. The "at IoU" (e.g., mAP @Shanky71.5, mAP@0.5:0.95) specifies the Intersection over Union (IoU) threshold or range for considering detections as true positives. IoU measures the overlap between the predicted and actual bounding boxes. Different IoU thresholds affect which detections are considered correct, thus indirectly influencing precision and recall and the shape of their curve.

For each class or at a collective level, as IoU thresholds vary, so do the associated precision and recall values for those conditions, leading to different mAP values.

In essence, as you vary IoU thresholds, the criteria for a detection being considered "correct" change, which then modifies the precision-recall relationship and, by extension, the mAP for that IoU setting.

I hope this helps clear things up! Feel free to ask if you have more questions.

[Shanky71]

@glenn-jocher Thanks for your reply.

1. I am still confused about the relationship between precision and recall. The yolov5 generates Precision and Recall values, I want to know at what decision threshold of Recall , Precision was calculated and at what decision threshold of Precision , Recall was calculated.

2. How is the precision-recall curve made? How we are able to get the multiple values of precision-recall that we are able to plot the curve. After Training, on the validation dataset we get a single value of precision-recall for all classes, but in the plot multiple values of precision-recall are being plotted? So how are these multiple values or precision-recall calculated? For ex. if we look at the precision-recall curve the PR coordinates (0.9,0.2) ; (0.88,0.4), (0.8,0.6) and so on. So how we are getting multiple values of Precision and recall? Is there any other variable (like confidence threshold) which is varying because of which multiple values of Precision and Recall?

3. False Positive (FP) is when the wrong detection is predicted by the model. The wrong detection can be either in the form of misclassification including negative label being labelled as positive label (a non-vehicle is labelled as a vehicle) or wrong bounding box . Which part of the FP is due to a wrong vehicle type (not so serious), or classifying no-vehicle objects as vehicles (serious).

[glenn-jocher]

Hey there! I'm here to help clarify your doubts. 😊

1. Precision and Recall Relationship: In YOLOv5, precision and recall are evaluated over a range of confidence thresholds, not a single "decision threshold." When you see a precision or recall value, it's typically at a specific confidence threshold, which ranges from 0 to 1. As the model predicts with different levels of confidence, precision and recall are calculated for each of these levels, which leads to the variety of points you see on a precision-recall (PR) curve.

2. How the Precision-Recall Curve is Made: The PR curve is plotted by calculating precision and recall at numerous confidence threshold levels. After training, when you evaluate the model on a validation dataset, the model predicts with a confidence score. By varying this confidence score from 0 to 1, you get different sets of precision and recall values. These points form the PR curve. So, the "other variable" you mentioned is indeed the confidence threshold.

3. Understanding False Positives (FP): False positives can indeed come in two flavors as you've noted: one where the object is correctly identified but assigned to the wrong class (vehicle type error), and another where a non-object (or an object from a non-target class) is falsely detected as an object of interest. Quantifying each type specifically requires analyzing the predictions and the ground truth. Typically, this analysis would involve reviewing the FP instances manually or setting up a more detailed evaluation to distinguish between these types of errors.

I hope this clears up your questions! If you need further clarification, feel free to ask. Happy detecting!

[Shanky71]

Thank you so much @glenn-jocher for helping me in clearing my doubts. Thanks for your time😊.

[glenn-jocher]

@Shanky71 you're welcome! 😊 I'm glad I could help. If you have any more questions or need further assistance, don't hesitate to reach out. Happy coding!

[Shanky71]

Hey @glenn-jocher I hope you are doing well. I have a small doubt about the objectness loss.

• I have divided my data in 80% training, 15% testing and 5% validation. As shown in the below image, what can be the possible reason of the sudden spike in the initial part and also, why there is higher objectness loss during validation than training.

• Also out of the three losses (class, objectness, box) which loss is more important. If I consider only class loss doesn't it also account for objectness and box loss?