search

ultralytics / ultralytics

Ultralytics YOLO11 🚀
https://docs.ultralytics.com
GNU Affero General Public License v3.0
32.62k stars 6.28k forks source link

RTDETR predict with TensorRT #8816

Closed Burhan-Q closed 5 months ago

Burhan-Q commented 8 months ago

Search before asking

YOLOv8 Component

Export

Bug

May be relevant to #6700 or #4237

No issue with exporting to TensorRT format with default rtdetr-l.pt pretrained model. Issue occurs during prediction. Seems that the class indices are incremented some how, but unfamiliar where this could occur.

Environment

Ultralytics YOLOv8.1.24 🚀 Python-3.10.12 torch-2.2.0+cu121 CUDA:0 (NVIDIA GeForce RTX 2060, 5924MiB)
Setup complete ✅ (12 CPUs, 15.6 GB RAM, 85.0/101.0 GB disk)

OS                  Linux-6.6.10-76060610-generic-x86_64-with-glibc2.35
Environment         Linux
Python              3.10.12
Install             pip
RAM                 15.56 GB
CPU                 AMD Ryzen 5 1600 Six-Core Processor
CUDA                12.1

matplotlib          ✅ 3.8.1>=3.3.0
opencv-python       ✅ 4.8.1.78>=4.6.0
pillow              ✅ 10.1.0>=7.1.2
pyyaml              ✅ 6.0.1>=5.3.1
requests            ✅ 2.31.0>=2.23.0
scipy               ✅ 1.11.3>=1.4.1
torch               ✅ 2.2.0>=1.8.0
torchvision         ✅ 0.17.0>=0.9.0
tqdm                ✅ 4.66.1>=4.64.0
psutil              ✅ 5.9.6
py-cpuinfo          ✅ 9.0.0
thop                ✅ 0.1.1-2209072238>=0.1.1
pandas              ✅ 2.1.3>=1.1.4
seaborn             ✅ 0.13.0>=0.11.0

Also checked on Windows but using experimental TensorRT install.

Minimal Reproducible Example


from ultralytics import YOLO, RTDETR

model = RTDETR("rtdetr-l.pt")
out = model.export(format="engine")

model = YOLO(out, task="detect")

# Error with logging
results = model.predict("ultralytics/assets/bus.jpg")

# Bypass verbose logging to get results
results = model.predict("ultralytics/assets/bus.jpg", verbose=False)

# Attempt plotting
plot = results[0].plot()
>>> Traceback (most recent call last):

>>>   File "<stdin>", line 1, in <module>
>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/model.py",
>>>   line 439, in predict
>>>     return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)

>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py",
>>>   line 206, in __call__
>>>     return list(self.stream_inference(source, model, *args, **kwargs))  # merge list of Result into one

>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/torch/utils/_contextlib.py",
>>>   line 35, in generator_context
>>>     response = gen.send(None)

>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py",
>>>   line 308, in stream_inference
>>>     s += self.write_results(i, self.results, (p, im, im0))

>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py",
>>>   line 173, in write_results
>>>     log_string += result.verbose()

>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/results.py",
>>>   line 327, in verbose
>>>     log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, "

>>> KeyError: 114

# Class indices too high, COCO pretrained model should only have max index of 79
results[0].boxes.cls
>>> tensor([114.,   0., 125., 220.,  14.,   5.,   4.], device='cuda:0')

Additional

Note pretrained RTDETR models can have names overridden with COCO class names


from ultralytics import YOLO, RTDETR

model = RTDETR("rtdetr-l.pt")
yolo = YOLO("yolov8n.pt")

# Set COCO class names
model.model.names = yolo.names.copy()

# Save copy with class names
model.save("rtdetr-l-names.pt")

# Load model with names
model = RTDETR("rtdetr-l-names.pt")

# Export
out = model.export(format="engine")

model = YOLO(out, task="detect")

# Same error as above
results = model.predict("ultralytics/assets/bus.jpg")

# Bypass verbose error
results = model.predict("ultralytics/assets/bus.jpg", verbose=False)

# Class names are loaded correctly
results[0].names
>>> {0: 'person', 1: 'bicycle', 2: 'car', 3: 'motorcycle', 4: 'airplane', 5: 'bus', 6: 'train', 7: 'truck', 8: 'boat',
 9: 'traffic light', 10: 'fire hydrant', 11: 'stop sign', 12: 'parking meter', 13: 'bench', 14: 'bird', 15: 'cat', 
16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe', 24: 'backpack', 
25: 'umbrella', 26: 'handbag', 27: 'tie', 28: 'suitcase', 29: 'frisbee', 30: 'skis', 31: 'snowboard', 32: 'sports ball', 
33: 'kite', 34: 'baseball bat', 35: 'baseball glove', 36: 'skateboard', 37: 'surfboard', 38: 'tennis racket', 
39: 'bottle', 40: 'wine glass', 41: 'cup', 42: 'fork', 43: 'knife', 44: 'spoon', 45: 'bowl', 46: 'banana', 47: 'apple', 
48: 'sandwich', 49: 'orange', 50: 'broccoli', 51: 'carrot', 52: 'hot dog', 53: 'pizza', 54: 'donut', 55: 'cake', 
56: 'chair', 57: 'couch', 58: 'potted plant', 59: 'bed', 60: 'dining table', 61: 'toilet', 62: 'tv', 63: 'laptop', 
64: 'mouse', 65: 'remote', 66: 'keyboard', 67: 'cell phone', 68: 'microwave', 69: 'oven', 70: 'toaster', 71: 'sink', 
72: 'refrigerator', 73: 'book', 74: 'clock', 75: 'vase', 76: 'scissors', 77: 'teddy bear', 78: 'hair drier', 
79: 'toothbrush'}

# Class indices not correct
results[0].boxes.cls
>>> tensor([114.,   0., 125., 220.,  14.,   5.,   4.], device='cuda:0')

# Same error as above
plot = results[0].plot()
>>> Traceback (most recent call last):
>>>   File "<stdin>", line 1, in <module>
>>>   File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/results.py", line 278, in plot
>>>     name = ("" if id is None else f"id:{id} ") + names[c]
>>> KeyError: 220

Are you willing to submit a PR?

Burhan-Q commented 8 months ago

Just tried again with ultralytics=8.1.27 after #8818 was merged

from ultralytics import YOLO, RTDETR

rtdetr = RTDETR()
# ensured that new model downloaded

rtdetr.names # all COCO names printed

out = rtdetr.export(format="engine", imgsz=640)

yolo = YOLO(out, task="detect")
results = yolo.predict("ultralytics/assets/bus.jpg")

Throws the following error

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>
  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/model.py", line 169, in __call__
    return self.predict(source, stream, **kwargs)

  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/model.py", line 439, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)

  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 168, in __call__
    return list(self.stream_inference(source, model, *args, **kwargs))  # merge list of Result into one

  File "/home/h4mmer/Documents/_code/ultra_testing/.ultra/lib/python3.10/site-packages/torch/utils/_contextlib.py",
   line 35, in generator_context
    response = gen.send(None)

  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 268,
   in stream_inference
    s[i] += self.write_results(i, Path(paths[i]), im, s)

  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/predictor.py", line 328, in write_results
    string += result.verbose() + f"{result.speed['inference']:.1f}ms"

  File "ultra_testing/.ultra/lib/python3.10/site-packages/ultralytics/engine/results.py", line 327, in verbose
    log_string += f"{n} {self.names[int(c)]}{'s' * (n > 1)}, "

KeyError: 91

...however, if I add to the predict arguments:

results = yolo.predict("ultralytics/assets/bus.jpg", verbose=False)

There is no error and the results are accessible as expected

results[0].boxes.data

>>> tensor([[0.0000e+00, 1.1440e+00, 0.0000e+00, 1.5867e+00, 8.7047e-01, 1.0100e+02],
            [0.0000e+00, 8.0206e-01, 0.0000e+00, 2.4063e+00, 8.5734e-01, 0.0000e+00],
            [0.0000e+00, 4.8752e-01, 0.0000e+00, 1.5062e+00, 7.2425e-01, 9.1000e+01],
            [0.0000e+00, 6.5691e-02, 0.0000e+00, 3.7359e-01, 7.1217e-01, 1.9900e+02],
            [0.0000e+00, 3.6141e-01, 0.0000e+00, 1.1519e+00, 4.5881e-01, 2.8000e+01],
            [0.0000e+00, 6.0656e-03, 0.0000e+00, 1.2300e-02, 3.7356e-01, 2.0000e+00]], device='cuda:0')

but the class indices are not correct.

results[0].boxes.cls
>>> tensor([101.,   0.,  91., 199.,  28.,   2.], device='cuda:0')
Burhan-Q commented 8 months ago

Well, that was silly, of course it shouldn't be loaded with YOLO it should be loaded with RT-DETR

from ultralytics import RTDETR

model = RTDETR("rtdetr-l.engine") # NOTE no task since only detect is supported

results = model.predict("bus.jpg")

results[0].boxes.data

>>> tensor([[ 6.6857e+02,  3.9463e+02,  8.0995e+02,  8.7887e+02,  9.5263e-01,  0.0000e+00],
            [ 1.3907e+01,  2.3246e+02,  8.0425e+02,  7.3299e+02,  9.5347e-01,  5.0000e+00],
            [ 4.9801e+01,  3.9784e+02,  2.4806e+02,  9.0482e+02,  9.5288e-01,  0.0000e+00],
            [ 2.2314e+02,  4.0515e+02,  3.4552e+02,  8.6061e+02,  9.2841e-01,  0.0000e+00],
            [ 3.9830e-01,  5.5183e+02,  7.7227e+01,  8.6938e+02,  8.8114e-01,  0.0000e+00],
            [-1.5663e-01,  2.5358e+02,  3.3768e+01,  3.2413e+02,  2.5948e-01,  1.1000e+01],
            [ 2.8695e+02,  4.8139e+02,  3.0145e+02,  5.2161e+02,  2.6323e-01,  2.7000e+01]], device='cuda:0')

results[0].boxes.cls

>>> tensor([ 0.,  5.,  0.,  0.,  0., 11., 27.], device='cuda:0')
hbl843449791 commented 7 months ago

I had the same problem, and using python calls solved the problem instead of the CLI

glenn-jocher commented 7 months ago

Great to hear that switching to Python calls solved the problem for you! 🎉 It's always good to have multiple approaches for troubleshooting. Just to make sure everyone can benefit from your solution, here's a quick code example demonstrating how to use Python calls for those who might be encountering similar issues:

from ultralytics import YOLO

# Load your model
model = YOLO('model_path_or_name.pt')

# Predict using the model
results = model.predict('path_to_image_or_video.jpg')

# Access predictions
print(results[0].pandas().xyxy)

If anyone has further questions or runs into more issues, feel free to ask! Happy coding! 😊

kalkun commented 6 months ago

The example in the docs using the CLI runs into the same key error issue: https://docs.ultralytics.com/models/rtdetr/#pre-trained-models as mentioned here. image

glenn-jocher commented 6 months ago

Hey there! 🌟 Thanks for bringing this to our attention. It looks like there might be a mismatch with class indices when using RTDETR models. While our team dives deeper into fixing this, you might want to try loading and predicting with the model directly in Python as a temporary workaround.

Here's a quick example on how to do it:

from ultralytics import RTDETR

# Load the model
model = RTDETR('rtdetr-l.pt')

# Run prediction
results = model.predict('path_to_your_image.jpg')

# You can print or plot the results
print(results[0].pandas().xyxy)

This approach should help bypass the key error issue for now. We appreciate your patience and are on it to get a fix out soon!

github-actions[bot] commented 5 months ago

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐