Batch Prediction with NCNNN

MABatin commented 4 months ago

Search before asking

[X] I have searched the YOLOv8 issues and found no similar bug report.

YOLOv8 Component

No response

Bug

I have exported yolov8n.pt model to NCNN framework with batched inference enabled but get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/engine/model.py", line 177, in __call__
    return self.predict(source, stream, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/engine/model.py", line 453, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/engine/predictor.py", line 168, in __call__
    return list(self.stream_inference(source, model, *args, **kwargs))  # merge list of Result into one
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/engine/predictor.py", line 255, in stream_inference
    self.results = self.postprocess(preds, im, im0s)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/models/yolo/detect/predict.py", line 25, in postprocess
    preds = ops.non_max_suppression(
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pi/easy-security_onprem/venv/lib/python3.11/site-packages/ultralytics/utils/ops.py", line 220, in non_max_suppression
    xc = prediction[:, 4:mi].amax(1) > conf_thres  # candidates
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: amax(): Expected reduction dim 1 to have non-zero size.

Environment

Ultralytics YOLOv8.2.28 🚀 Python-3.11.2 torch-2.1.1 CPU (Cortex-A72) Setup complete ✅ (4 CPUs, 3.7 GB RAM, 10.7/58.0 GB disk)

Minimal Reproducible Example

model.export(format="ncnn", batch=4, half=True)
ncnn_model = YOLO("weights/yolov8n_ncnn_model")
results = ncnn_model(["https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/bus.jpg", "https://ultralytics.com/images/bus.jpg"], half=True)

Additional

No response

Are you willing to submit a PR?

[ ] Yes I'd like to help by submitting a PR!

glenn-jocher commented 3 months ago

@MABatin hello! Thank you for providing detailed information about the issue you're encountering with batch prediction using the NCNN framework.

The error you're seeing, IndexError: amax(): Expected reduction dim 1 to have non-zero size, typically indicates that the predictions tensor is empty. This can happen if the model does not detect any objects in the input images, possibly due to the confidence threshold being too high or other preprocessing steps affecting the visibility of detectable objects.

Here are a few steps you can try to resolve this issue:

Check Input Images: Ensure the images you're using contain clear, detectable objects. You might want to test with different images known to contain easily detectable objects.
Adjust Confidence Threshold: Lower the confidence threshold when calling the model to see if it starts detecting objects.
Review Preprocessing: Ensure that the preprocessing steps (like resizing and normalization) are appropriate for the NCNN model.

If these steps do not resolve the issue, it would be helpful to look into the model's detailed output right before the non-max suppression step to see if the model is producing any predictions at all.

Please let us know if these suggestions help or if you need further assistance!

github-actions[bot] commented 2 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Docs: https://docs.ultralytics.com
HUB: https://hub.ultralytics.com
Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

ultralytics / ultralytics