Is there a script to test the mAP of the engine file?

[shihanyu]

🚀 Feature

 When a model file is transferred into tensorrt format (.pt to .engine ), the performance may be infulenced. so is there a test file to culculate the mAP with the engine file?

Motivation

Pitch

Alternatives

Additional context

[github-actions[bot]]

👋 Hello @shihanyu, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide screenshots and minimum viable code to reproduce your issue, otherwise we can not help you.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset images, training logs, screenshots, and a public link to online W&B logging if available.

For business inquiries or professional support requests please visit https://ultralytics.com or email Glenn Jocher at glenn.jocher@ultralytics.com.

Requirements

Python>=3.6.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

$ git clone https://github.com/ultralytics/yolov5
$ cd yolov5
$ pip install -r requirements.txt

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Google Colab and Kaggle notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training (train.py), validation (val.py), inference (detect.py) and export (export.py) on MacOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

@shihanyu no there is not. val.py will only produce metrics on PyTorch models currently.

Detect.py will run inference on PyTorch, ONNX runtime, ONNX OpenCV DNN, TorchScript, TFLite models, TF pb and TF saved_model formats.

[francescotaioli]

@shihanyu coco eval api could be used to get the mAP. In order to do so, you have to create the json (in the coco format) that contain the prediction. You can create this json using the output from the engine. See also https://stackoverflow.com/a/69556204/5723524

[glenn-jocher]

@shihanyu good news 😃! Your original issue may now be partially addressed ✅ in PR #5549. This PR consolidates all backends into a single YOLOv5 DetectMultiBackend() class. Supports following model formats:

• [x] PyTorch *.pt
• [x] TorchScript *.torchscript.pt
• [x] CoreML *.mlmodel
• [x] TensorFlow saved_model
• [x] TensorFlow *.pb
• [x] TensorFlow Lite *.tflite
• [ ] TensorFlow.js web_model
• [x] ONNX Runtime *.onnx
• [x] OpenCV DNN *.onnx

TensorRT is not currently supported, but if you have experience with it please consider submitting a PR to help others run inference with it.

Usage

# Export
python export.py --weights yolov5s.pt --include tflite

# Inference
python detect.py --weights yolov5s.tflite
python val.py --weights yolov5s.tflite

To receive this update:

• Git – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebooks – View updated notebooks
• Docker – sudo docker pull ultralytics/yolov5:latest to update your image

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

[shihanyu]

Hi, I have solved the problem by combine the test.py in yolov5 and the yolov5trt.py in tensorrtx,so that it can validate the .engine file. And if any of you want the file, I will give my file to you .

从 Windows 版邮件https://go.microsoft.com/fwlink/?LinkId=550986发送

发件人: Glenn @.> 发送时间: 2021年11月10日 6:03 收件人: @.> 抄送: @.>; @.> 主题: Re: [ultralytics/yolov5] Is there a script to test the mAP of the engine file? (Issue #5278)

@shihanyuhttps://github.com/shihanyu good news 😃! Your original issue may now be partially addressed ✅ in PR #5549https://github.com/ultralytics/yolov5/pull/5549. This PR consolidates all backends into a single YOLOv5 DetectMultiBackend() class. Supports following model formats:

• [X] PyTorch *.pt
• [X] TorchScript *.torchscript.pt
• [X] CoreML *.mlmodel
• [X] TensorFlow saved_model
• [X] TensorFlow *.pb
• [X] TensorFlow Lite *.tflite
• [ ] TensorFlow.js web_model
• [X] ONNX Runtime *.onnx
• [X] OpenCV DNN *.onnx

TensorRT is not currently supported, but if you have experience with it please consider submitting a PR to help others run inference with it.

Usage

Export

python export.py --weights yolov5s.pt --include tflite

Inference

python detect.py --weights yolov5s.tflite

python val.py --weights yolov5s.tflite

To receive this update:

• Githttps://github.com/ultralytics/yolov5 – git pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
• PyTorch Hubhttps://pytorch.org/hub/ultralytics_yolov5/ – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
• Notebookshttps://github.com/ultralytics/yolov5/blob/master/tutorial.ipynb – View updated notebooks [Open In Colab] https://colab.research.google.com/github/ultralytics/yolov5/blob/master/tutorial.ipynb [Open In Kaggle] https://www.kaggle.com/ultralytics/yolov5
• Dockerhttps://hub.docker.com/r/ultralytics/yolov5 – sudo docker pull ultralytics/yolov5:latest to update your image [Docker Pulls] https://hub.docker.com/r/ultralytics/yolov5

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ultralytics/yolov5/issues/5278#issuecomment-964587070, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE546SSRCR722BW3PUD5XHTULGLCBANCNFSM5GNU7Z5Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[glenn-jocher]

@shihanyu hmm interesting. Maybe I'll contact the trtx author to see if he'd help us with a PR.

[shihanyu]

I will be very glad if I can help. Looking forward to your good news.

从 Windows 版邮件https://go.microsoft.com/fwlink/?LinkId=550986发送

发件人: Glenn @.> 发送时间: 2021年11月10日 18:58 收件人: @.> 抄送: @.>; @.> 主题: Re: [ultralytics/yolov5] Is there a script to test the mAP of the engine file? (Issue #5278)

@shihanyuhttps://github.com/shihanyu hmm interesting. Maybe I'll contact the trtx author to see if he'd help us with a PR.

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ultralytics/yolov5/issues/5278#issuecomment-965017215, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE546SXCS34GCMNHXWKJXKTULJF43ANCNFSM5GNU7Z5Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[shihanyu]

Hi, I just apply the merge request for the function to validate the tensorrt model file. Hope it helps, let me know if it works or not.

Best Regards

Hanyu

从 Windows 版邮件https://go.microsoft.com/fwlink/?LinkId=550986发送

发件人: Glenn @.> 发送时间: 2021年11月10日 18:58 收件人: @.> 抄送: @.>; @.> 主题: Re: [ultralytics/yolov5] Is there a script to test the mAP of the engine file? (Issue #5278)

@shihanyuhttps://github.com/shihanyu hmm interesting. Maybe I'll contact the trtx author to see if he'd help us with a PR.

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ultralytics/yolov5/issues/5278#issuecomment-965017215, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE546SXCS34GCMNHXWKJXKTULJF43ANCNFSM5GNU7Z5Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[glenn-jocher]

@shihanyu great, I will take a look today!

[shihanyu]

Hi, does the script work ? Do I need to reform my code? Maybe ajust the structure?

从 Windows 版邮件https://go.microsoft.com/fwlink/?LinkId=550986发送

发件人: Glenn @.> 发送时间: 2021年11月12日 19:25 收件人: @.> 抄送: @.>; @.> 主题: Re: [ultralytics/yolov5] Is there a script to test the mAP of the engine file? (Issue #5278)

@shihanyuhttps://github.com/shihanyu great, I will take a look today!

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ultralytics/yolov5/issues/5278#issuecomment-967027717, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE546SWXOKYIQ6XBMQEERV3ULT2RTANCNFSM5GNU7Z5Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[shihanyu]

Hi Glenn My requrest for merge failed because it can not install pycuda …，any advice for that ?

Best regards hanyu

从 Windows 版邮件https://go.microsoft.com/fwlink/?LinkId=550986发送

---

发件人: Glenn Jocher @.> 发送时间: Friday, November 12, 2021 7:25:13 PM 收件人: ultralytics/yolov5 @.> 抄送: shihanyu @.>; Mention @.> 主题: Re: [ultralytics/yolov5] Is there a script to test the mAP of the engine file? (Issue #5278)

@shihanyuhttps://github.com/shihanyu great, I will take a look today!

― You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/ultralytics/yolov5/issues/5278#issuecomment-967027717, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AE546SWXOKYIQ6XBMQEERV3ULT2RTANCNFSM5GNU7Z5Q. Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[glenn-jocher]

@shihanyu yes, all 3rd party dependencies that are not part of requirements.txt can be placed in check_requirements() calls only if/when they are actually used. For example ONNX inference first checks to see if onnx and onnxruntime are installed here before importing them. If a user never uses ONNX models, these packages are never installed. https://github.com/ultralytics/yolov5/blob/e80a09bbfa1ddb1097fdc7164d84dedeb3d95388/models/common.py#L315-L319

[github-actions[bot]]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!