NEW - YOLOv8 🚀 Multi-Object Tracking

[glenn-jocher]

YOLOv8 Multi-Object Tracking

Object tracking is a task that involves identifying the location and class of objects, then assigning a unique ID to that detection in video streams.

The output of tracker is the same as detection with an added object ID.

Available Trackers

The following tracking algorithms have been implemented and can be enabled by passing tracker=tracker_type.yaml

• BoT-SORT - botsort.yaml
• ByteTrack - bytetrack.yaml

The default tracker is BoT-SORT.

Tracking

Use a trained YOLOv8n/YOLOv8n-seg model to run tracker on video streams.

Python

from ultralytics import YOLO

# Load a model
model = YOLO("yolov8n.pt")  # load an official detection model
model = YOLO("yolov8n-seg.pt")  # load an official segmentation model
model = YOLO("path/to/best.pt")  # load a custom model

# Track with the model
results = model.track(source="https://youtu.be/Zgi9g1ksQHc", show=True) 
results = model.track(source="https://youtu.be/Zgi9g1ksQHc", show=True, tracker="bytetrack.yaml") 

CLI

yolo track model=yolov8n.pt source="https://youtu.be/Zgi9g1ksQHc"  # official detection model
yolo track model=yolov8n-seg.pt source=...   # official segmentation model
yolo track model=path/to/best.pt source=...  # custom model
yolo track model=path/to/best.pt  tracker="bytetrack.yaml" # bytetrack tracker

As in the above usage, we support both the detection and segmentation models for tracking and the only thing you need to do is loading the corresponding (detection or segmentation) model.

Configuration

Tracking

Tracking shares the configuration with predict, i.e conf, iou, show. More configurations please refer to predict page. !!! example ""

Python

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model.track(source="https://youtu.be/Zgi9g1ksQHc", conf=0.3, iou=0.5, show=True) 

CLI

yolo track model=yolov8n.pt source="https://youtu.be/Zgi9g1ksQHc" conf=0.3, iou=0.5 show

Tracker

We also support using a modified tracker config file, just copy a config file i.e. custom_tracker.yaml from ultralytics/tracker/cfg and modify any configurations(expect the tracker_type) you need to.

Python

from ultralytics import YOLO

model = YOLO("yolov8n.pt")
results = model.track(source="https://youtu.be/Zgi9g1ksQHc", tracker='custom_tracker.yaml') 

CLI

yolo track model=yolov8n.pt source="https://youtu.be/Zgi9g1ksQHc" tracker='custom_tracker.yaml'

Please refer to ultralytics/tracker/cfg page.

[akashAD98]

@glenn-jocher its supporting .onnx, .trt or openvino.xml weights for tracking? instead of only .pt weight

[Laughing-q]

@akashAD98 I haven't tested yet but technically track mode supports whatever format predict mode supports. So yes it's supporting .onnx, .trt and other formats.

[glenn-jocher]

@akashAD98 @Laughing-q yes that's right! Tracking supports any predict or segment models in any of the following formats (TF.js is not supported for inference, but all other formats are).

Available YOLOv8 export formats are in the table below. You can predict, track or val directly on exported models, i.e. yolo track model=yolov8n.onnx.

Format	format Argument	Model	Metadata
PyTorch	-	yolov8n.pt	✅
TorchScript	torchscript	yolov8n.torchscript	✅
ONNX	onnx	yolov8n.onnx	✅
OpenVINO	openvino	yolov8n_openvino_model/	✅
TensorRT	engine	yolov8n.engine	✅
CoreML	coreml	yolov8n.mlmodel	✅
TF SavedModel	saved_model	yolov8n_saved_model/	✅
TF GraphDef	pb	yolov8n.pb	❌
TF Lite	tflite	yolov8n.tflite	✅
TF Edge TPU	edgetpu	yolov8n_edgetpu.tflite	✅
TF.js	tfjs	yolov8n_web_model/	✅
PaddlePaddle	paddle	yolov8n_paddle_model/	✅

[glenn-jocher]

@zldrobit I've managed to include metadata additions into all YOLOv8 model formats above except for TF *.pb models. Do you know if this is possible for this format? The metadata is a dictionary here. For directory exports I simply place a metadata.yaml inside the directory, for file formats like ONNX I've found methods to embed the metadata dict inside the file.

https://github.com/ultralytics/ultralytics/blob/30fc4b537ff1d9b115bc1558884f6bc2696a282c/ultralytics/yolo/engine/exporter.py#L214-L224

[akashAD98]

@akashAD98 @Laughing-q yes that's right! Tracking supports any predict or segment models in any of the following formats (TF.js is not supported for inference, but all other formats are).

Available YOLOv8 export formats are in the table below. You can predict, track or val directly on exported models, i.e. yolo track model=yolov8n.onnx.

Format format Argument Model Metadata PyTorch - yolov8n.pt ✅ TorchScript torchscript yolov8n.torchscript ✅ ONNX onnx yolov8n.onnx ✅ OpenVINO openvino yolov8n_openvino_model/ ✅ TensorRT engine yolov8n.engine ✅ CoreML coreml yolov8n.mlmodel ✅ TF SavedModel saved_model yolov8n_saved_model/ ✅ TF GraphDef pb yolov8n.pb ❌ TF Lite tflite yolov8n.tflite ✅ TF Edge TPU edgetpu yolov8n_edgetpu.tflite ✅ TF.js tfjs yolov8n_web_model/ ✅ PaddlePaddle paddle yolov8n_paddle_model/ ✅

if i pass openvino weights, its not support

model = YOLO("yolov8n.xml")

should i need to do different processing? @glenn-jocher

[glenn-jocher]

@akashAD98 your openvino usage is not aligned with the usage example we've shown (and that's you've pasted).

[akashAD98]

@akashAD98 your openvino usage is not aligned with the usage example we've shown (and that's you've pasted).

yes got it thanks

[akashAD98]

@glenn-jocher tracker is not working for custom trained models, I have trained the model,i have 20 classes & simply passed them to tracker & its gives error, while default weights are working fine

video 1/1 (1/1517) /home/ak/s/v8/notebook_office.mp4: 384x640 1 blackboard, 66.3ms
Traceback (most recent call last):
  File "yolov8_tracker.py", line 4, in <module>
    results = model.track(data='custom_data.yaml',source="notebook_office.mp4", conf=0.5, iou=0.5, show=False,device='CPU',project='anushka_yolov8Nano50conf',save=True)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/yolo/engine/model.py", line 236, in track
    return self.predict(source=source, stream=stream, **kwargs)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/yolo/engine/model.py", line 227, in predict
    return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/yolo/engine/predictor.py", line 114, in __call__
    return list(self.stream_inference(source, model))  # merge list of Result into one
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 64, in generator_context
    response = gen.send(request)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/yolo/engine/predictor.py", line 177, in stream_inference
    self.run_callbacks('on_predict_postprocess_end')
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/yolo/engine/predictor.py", line 267, in run_callbacks
    callback(self)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/tracker/track.py", line 35, in on_predict_postprocess_end
    tracks = predictor.trackers[i].update(det, im0s[i])
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/tracker/trackers/byte_tracker.py", line 212, in update
    warp = self.gmc.apply(img, dets)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/tracker/utils/gmc.py", line 78, in apply
    return self.applySparseOptFlow(raw_frame, detections)
  File "/home/refx/DS/yolov7_tracker_yono/openvino_env/lib/python3.8/site-packages/ultralytics/tracker/utils/gmc.py", line 272, in applySparseOptFlow
    matchedKeypoints, status, err = cv2.calcOpticalFlowPyrLK(self.prevFrame, frame, self.prevKeyPoints, None)
cv2.error: OpenCV(4.7.0) /io/opencv/modules/video/src/lkpyramid.cpp:1260: error: (-215:Assertion failed) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function 'calc'

command im using

from ultralytics import YOLO

model = YOLO("best_custom20_yolov8s.pt")
results = model.track(data='custom_data.yaml',source="notebook_office.mp4", conf=0.5, iou=0.5, show=False,device='CPU',project='opvidetracker',save=True)

[Laughing-q]

@akashAD98 I just tested with a custom trained model that detect human-head and it works fine me, no errors

from ultralytics import YOLO

model = YOLO("best.pt")
results = model.track(
    source="test.mp4",
    conf=0.5,
    iou=0.5,
    show=False,
    device="CPU",
    save=True,
)

your error seems like a cv2 issue that related to the gmc module in trakcer, it's probably related to your detected boxes and original frame.

[akashAD98]

@Laughing-q i tried to install ultrlytics & defult its using opencv, also, i tested my weight on google collab,still, I'm getting the same error for yolov8s,yolov8m its working fine.but for custom getting cv2 error

which OpenCV version i need to install,

error: OpenCV(4.7.0) /io/opencv/modules/video/src/lkpyramid.cpp:1260: error: (-215:Assertion failed) (npoints = prevPtsMat.checkVector(2, CV_32F, true)) >= 0 in function 'calc'

[Laughing-q]

@akashAD98 I suppose the 4.7.0 in your error log is the version right? I'm also using this version. We use bot-sort tracker as default which includes the gmc module but bytetrack do not have it. For this case that I'm not able to reproduce the error, I suggest you to use btyetrack for now.

[akashAD98]

@Laughing-q thanks its working fine for bytetracker, for bot tracker im getting that error

also i have one quetions, by defult its taking all files from ultrlytics YOLO if i want to modify any files, shoul i directly upload that file there & give path of that file? im correct

for the custom model, I need to pass data.yaml file which contains names, for tracker also i can do the same

data.yaml file has my custom names of classes model.track(data='data.yaml',tracker='bytetracker.yaml')

[zldrobit]

@glenn-jocher

@zldrobit I've managed to include metadata additions into all YOLOv8 model formats above except for TF *.pb models. Do you know if this is possible for this format? The metadata is a dictionary here. For directory exports I simply place a metadata.yaml inside the directory, for file formats like ONNX I've found methods to embed the metadata dict inside the file.

https://github.com/ultralytics/ultralytics/blob/30fc4b537ff1d9b115bc1558884f6bc2696a282c/ultralytics/yolo/engine/exporter.py#L214-L224

Sorry for the late reply. To the best of my knowledge, a GraphDef .pb file does not include any meta information. It contains only the computation graph (network structure) and the name/weights of each node. There's no official tutorial of TensorFlow to add metadata in GraphDef .pb files. However, it is possible to use protobuf api pb.MergeFrom(metadata) to merge metadata into pb (https://googleapis.dev/python/protobuf/latest/google/protobuf/any_pb2.html#google.protobuf.any_pb2.Any.MergeFrom).

[Laughing-q]

@akashAD98 well you don't need to pass data to get the names when you do tracking, names would be remembered as an attribute of model while training and it would be read from model when you do tracking.

[mohamedamine99]

Greetings,

I have a question since this is my first time working with object trackers in general : Can YOLOv8 builtin trackers be used for multi-object tracking on video frames read by OpenCV?

I am trying to use the YOLOv8 Builtin Tracker for multi-object tracking, but I am unsure if it is possible to use it on video frames read one-by-one from OpenCV using cap.read() instead of a pre-existing full video or video stream.

I have searched the documentation and the GitHub repository for YOLOv8, but I could not find any information on this topic. I would appreciate it if you could clarify whether it is possible to use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV.

Thank you for your time and attention. I look forward to your responses.

[glenn-jocher]

Hello! Yes, you can use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV. The tracker can be initialized on a single frame and then updated on subsequent frames.

Here is a brief overview of how you can do it:

1. Initialize the detector and the tracker
2. Read a frame from OpenCV using cap.read()
3. Pass the frame through the detector and get the detections
4. Pass the detections to the tracker and update the tracks
5. Repeat steps 2-4 for each frame

Here is some sample code to get you started:

import cv2
from yolov5 import Detector
from yolov5.utils.bbox import xyxy2xywh
from yolov5.utils.tracker import Tracker

# Initialize the detector and the tracker
detector = Detector(weights='yolov5s.pt')
tracker = Tracker(threshold=0.5)

# Open the video capture
cap = cv2.VideoCapture('test.mp4')
while cap.isOpened():
    # Read a frame
    ret, frame = cap.read()
    if not ret:
        break

    # Pass the frame through the detector and get the detections
    detections = detector.detect(frame)

    # Pass the detections to the tracker and update the tracks
    tracker.update(xyxy2xywh(detections))

    # Draw the tracks on the frame
    for track in tracker.tracks:
        cv2.rectangle

[mohamedamine99]

Hello! Yes, you can use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV. The tracker can be initialized on a single frame and then updated on subsequent frames.

Here is a brief overview of how you can do it:

1. Initialize the detector and the tracker
2. Read a frame from OpenCV using cap.read()
3. Pass the frame through the detector and get the detections
4. Pass the detections to the tracker and update the tracks
5. Repeat steps 2-4 for each frame

Here is some sample code to get you started:

import cv2
from yolov5 import Detector
from yolov5.utils.bbox import xyxy2xywh
from yolov5.utils.tracker import Tracker

# Initialize the detector and the tracker
detector = Detector(weights='yolov5s.pt')
tracker = Tracker(threshold=0.5)

# Open the video capture
cap = cv2.VideoCapture('test.mp4')
while cap.isOpened():
    # Read a frame
    ret, frame = cap.read()
    if not ret:
        break

    # Pass the frame through the detector and get the detections
    detections = detector.detect(frame)

    # Pass the detections to the tracker and update the tracks
    tracker.update(xyxy2xywh(detections))

    # Draw the tracks on the frame
    for track in tracker.tracks:
        cv2.rectangle

Hi Glenn , thank you for your quick response. I appreciate your help. However, I noticed that the code you provided is for YOLOv5, while I am specifically looking to use the YOLOv8 Builtin Tracker. Is there an example with YOLOv8 trackers using the track() method as in model.track(....) ?

[glenn-jocher]

@mohamedamine99 yes, apologies for the confusion please see https://docs.ultralytics.com/modes/track for Python tracker usage :)

[mohamedamine99]

@mohamedamine99 yes, apologies for the confusion please see https://docs.ultralytics.com/modes/track for Python tracker usage :)

thanks, I'll check it out

[glenn-jocher]

You're welcome! Let us know if you have any further questions or concerns. We're always here to help.

[akashAD98]

@glenn-jocher its possible to use tracker using cv2 ,instead of directly using model.track() which is already mentioned in document.

I want to print & get detail of each bounding box,score,label from tracker .using cv2 method. Thanks

[akashAD98]

@glenn-jocher thanks same thing exactly i want to use it for yolov8 model & there tracker bot ,bytetravker . If I replace yolov5 with yolov8 weights will it work?

[glenn-jocher]

Yes, if you replace the YOLOv5 weights with YOLOv8 weights, the model architecture should remain the same and the model should work as expected with the YOLOv8 weights. However, keep in mind that the performance and accuracy of the model may differ when using different weights. Also, make sure that the input image size and other hyperparameters are adjusted accordingly when switching between models.

[glenn-jocher]

@RwGrid ah it's simpler than that, you can use model.track(), and if you want to specify a tracker you can use the tracker arg:

from ultralytics import YOLO

model = YOLO('yolov8n.pt')
results = model.track(source='video.mp4', tracker='bot_sort.yaml', stream=True)

for result in results:
    # process results...

[RwGrid]

ultralytics-8.0.58 works, it works. may the (shadow of bugs )never follow you, sir, thank you

[glenn-jocher]

@RwGrid you're welcome! I'm glad that it worked for you. Feel free to reach out to us if you have any more questions or concerns.

[vaventt]

@glenn-jocher I have a problem, can you please help. From back-end by API I'm getting frames(that were extracted from video stream) in bytes format, than converting in into suitable for YOLOv8 format and inputting to the source argument, I want YOLOv8 tracker to remember cars with their respective unique IDs, for now running this code, I'm getting same IDs for each frame, they are not incrementing in the way, when you input as source video or stream, can I solve this issue using this simplified .track() method? Or for this I would need to import and build my custom tracker and using YOLOv8 only as object detector with .predict() method? I would like to keep code as simple as possible and stick to the .track() method instead of creating hundreds of rows with separate tracker algorithm.

@app.post("/objectdetection")
async def detect_cars_return_img(file: bytes = File(...)):
    input_image = Image.open(io.BytesIO(file)).convert("RGB")
    results = model.track(source=input_image, classes=[2, 5, 7])
    res_plotted = results[0].plot()

    success, encoded_image = cv2.imencode('.jpg', res_plotted)
    content = encoded_image.tobytes()
    print(content == file)
    return Response(content=content, media_type="image/jpg")

[Laughing-q]

@Andrii-DataScientist I made a PR to allow this usage. It'll use the shared tracker all the time with global_tracker=True. :) #1887

[SOUMIA21]

Hello! Yes, you can use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV. The tracker can be initialized on a single frame and then updated on subsequent frames.

Here is a brief overview of how you can do it:

1. Initialize the detector and the tracker
2. Read a frame from OpenCV using cap.read()
3. Pass the frame through the detector and get the detections
4. Pass the detections to the tracker and update the tracks
5. Repeat steps 2-4 for each frame

Here is some sample code to get you started:

import cv2
from yolov5 import Detector
from yolov5.utils.bbox import xyxy2xywh
from yolov5.utils.tracker import Tracker

# Initialize the detector and the tracker
detector = Detector(weights='yolov5s.pt')
tracker = Tracker(threshold=0.5)

# Open the video capture
cap = cv2.VideoCapture('test.mp4')
while cap.isOpened():
    # Read a frame
    ret, frame = cap.read()
    if not ret:
        break

    # Pass the frame through the detector and get the detections
    detections = detector.detect(frame)

    # Pass the detections to the tracker and update the tracks
    tracker.update(xyxy2xywh(detections))

    # Draw the tracks on the frame
    for track in tracker.tracks:
        cv2.rectangle

If I want to use the same code but the predictions and the tracking results are based on yolov8 and what can I extract from this line : results = model.track(source="https://youtu.be/Zgi9g1ksQHc", tracker='custom_tracker.yaml')

[glenn-jocher]

@SOUMIA21 yes, you can use the same code to perform multi-object tracking with YOLOv8. You will need to update the code to use the YOLOv8 model for detection instead of YOLOv5, and make any necessary adjustments to the input image size and other hyperparameters. The results object returned by model.track() will contain the tracked objects with corresponding IDs, so you can use that to access the tracking results.

[vaventt]

@Laughing-q thank you so much for such a quick solution, but somehow I can't track objects with this argument persist=True, with the first image everything goes fine on second it always returns an error. I tried to reinstall opencv-python and I have version that satisfies YOLOv8 requirements, but still get this error, tried even in Jupiter with simple example, again the same problem, tried to find out a solution to this error on StackOverflow, but couldn't, can you help me? With tracker='bytetrack.yaml there isn't such problem, earlier I didn't get this error, until applying persist parameter.

cv2.error: OpenCV(4.7.0) /Users/runner/work/opencv-python/opencv-python/opencv/modules/video/src/lkpyramid.cpp:1394: error: (-215:Assertion failed) prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size() in function 'calc'

My code example:

from ultralytics import YOLO
from PIL import Image
model = YOLO("yolov8n.pt")
im1 = Image.open('traffic.jpg').convert('RGB')
results = model.track(source=im1, persist=True, classes=[2,5,7])
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

im2 = Image.open('traffic2.jpg').convert('RGB')
results = model.track(source=im2, persist=True, classes=[2,5,7])
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

[vaventt]

@@Laughing-q and one more, do you have any resources or links to them with examples of how to deploy YOLOv8 with docker and fastapi? Locally my fastapi working great, now with persist=True and tracker=bytetrack.yaml I have got what I wanted, last what I need is to wrap my python file into docker container, it builds, it launches but then I got always problems with lap library, with which many people had problems as I searched through issues, on my macOS everything was flawless, but with docker it just don't want to work, when I input my first image through API I always get an error related to lap, RUN pip install lap, RUN pip install git+git://github.com/gatagat/lap.git, many other tries have failed, first I had a problem can't import lap library then with RUN git clone https://github.com/gatagat/lap.git I could overcome this problem and now I again get new errors, what interesting on first image model works, but on next one it just brokes

  File "/usr/local/lib/python3.8/site-packages/ultralytics/tracker/utils/matching.py", line 53, in linear_assignment
    _, x, y = lap.lapjv(cost_matrix, extend_cost=True, cost_limit=thresh)
AttributeError: module 'lap' has no attribute 'lapjv'

[glenn-jocher]

@vaventt It seems that the lap module is causing problems when you try to deploy your YOLOv8 model with Docker and FastAPI. One possible reason for this is that the lap module is not properly installed in the Docker container.

To resolve this problem, you can try installing lap using pip install lap==0.4.0. If that doesn't work, you can also try installing lap from the GitHub repository using git clone https://github.com/gatagat/lap.git and then pip install -e lap/.

In terms of deploying YOLOv8 with Docker and FastAPI, you can check out the following resources for guidance:

• Dockerizing a FastAPI app: https://fastapi.tiangolo.com/deployment/docker/
• Building a container for running YOLOv5 on CPU: https://github.com/ultralytics/yolov5/wiki

[SOUMIA21]

@glenn-jocher Sorry to bother you again. but really I don't know how to get information from the variable results of this line ( results = model.track())

1. when I try to use results[0].xyxy I got an error because it's a generator
2. I tried to convert results to a list but the function list() saves all the information on one row and I don't know how to recover parameters one by one (xyxy, conf, scores...).
3. I found in this discussion that @vaventt is using (results[0].plot() ). Could you please send me the link if there is any detailed doc of how can I use the results of this track() (rather than this one: https://docs.ultralytics.com/modes/track/#tracking_1)
4. @glenn-jocher @Laughing-q if you are still developing the SW of this track please just inform us to wait for it.

Thanks again for your quick support, I appreciate it a lot

Hello! Yes, you can use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV. The tracker can be initialized on a single frame and then updated on subsequent frames. Here is a brief overview of how you can do it:

1. Initialize the detector and the tracker
2. Read a frame from OpenCV using cap.read()
3. Pass the frame through the detector and get the detections
4. Pass the detections to the tracker and update the tracks
5. Repeat steps 2-4 for each frame

Here is some sample code to get you started:

import cv2
from yolov5 import Detector
from yolov5.utils.bbox import xyxy2xywh
from yolov5.utils.tracker import Tracker

# Initialize the detector and the tracker
detector = Detector(weights='yolov5s.pt')
tracker = Tracker(threshold=0.5)

# Open the video capture
cap = cv2.VideoCapture('test.mp4')
while cap.isOpened():
    # Read a frame
    ret, frame = cap.read()
    if not ret:
        break

    # Pass the frame through the detector and get the detections
    detections = detector.detect(frame)

    # Pass the detections to the tracker and update the tracks
    tracker.update(xyxy2xywh(detections))

    # Draw the tracks on the frame
    for track in tracker.tracks:
        cv2.rectangle

If I want to use the same code but the predictions and the tracking results are based on yolov8 and what can I extract from this line : results = model.track(source="https://youtu.be/Zgi9g1ksQHc", tracker='custom_tracker.yaml')

@glenn-jocher Sorry to bother you again. but really I don't know how to get information from the variable results of this line ( results = model.track())

1. when I try to use results[0].xyxy I got an error because it's a generator
2. I tried to convert results to a list but the function list() saves all the information on one row and I don't know how to recover parameters one by one (xyxy, conf, scores...).
3. I found in this discussion that @vaventt is using (results[0].plot() ). Could you please send me the link if there is any detailed doc of how can I use the results of this track() (rather than this one: https://docs.ultralytics.com/modes/track/#tracking_1)
4. @glenn-jocher @Laughing-q if you are still developing the SW of this track please just inform us to wait for it.

Thanks again for your quick support, I appreciate it a lot

[Laughing-q]

@Laughing-q thank you so much for such a quick solution, but somehow I can't track objects with this argument persist=True, with the first image everything goes fine on second it always returns an error. I tried to reinstall opencv-python and I have version that satisfies YOLOv8 requirements, but still get this error, tried even in Jupiter with simple example, again the same problem, tried to find out a solution to this error on StackOverflow, but couldn't, can you help me? With tracker='bytetrack.yaml there isn't such problem, earlier I didn't get this error, until applying persist parameter.

cv2.error: OpenCV(4.7.0) /Users/runner/work/opencv-python/opencv-python/opencv/modules/video/src/lkpyramid.cpp:1394: error: (-215:Assertion failed) prevPyr[level * lvlStep1].size() == nextPyr[level * lvlStep2].size() in function 'calc'

My code example:

from ultralytics import YOLO
from PIL import Image
model = YOLO("yolov8n.pt")
im1 = Image.open('traffic.jpg').convert('RGB')
results = model.track(source=im1, persist=True, classes=[2,5,7])
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

im2 = Image.open('traffic2.jpg').convert('RGB')
results = model.track(source=im2, persist=True, classes=[2,5,7])
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

I'm not able to reproduce the issue...the script works fine in my case.

[Laughing-q]

@SOUMIA21 you can access xyxy, conf, cls with model.track just like model.predict. https://docs.ultralytics.com/modes/predict/

results = model.track(source=..., stream=True)
for result in results:
    boxes = result.boxes  # Boxes object for bbox outputs
    masks = result.masks  # Masks object for segmentation masks outputs
    probs = result.probs  # Class probabilities for classification outputs
    boxes.xyxy
    boxes.conf
    boxes.cls
    boxes.id

[SOUMIA21]

For tracking objects in a video

1. using this tracker on a set of frames works just as a model for prediction; all the time gives NONE for IDs However, using the tracker with a single image works very well. ==> Is there any way to force the tracker to give a Ids to the predictions

2. and using the method proposed by @glenn-jocher , how can I do this step (Pass the detections to the tracker and update the tracks) with this tracker

3. I used the tracker the detector and I tried to with it only, and I have the same problem I can't get IDs.

[glenn-jocher]

@SOUMIA21 1. The YOLOv8 tracker is designed to track objects between frames in a video. As such, it relies on the detections made in each frame being associated with the same object across multiple frames. This is usually achieved by using an ID for each detected object, based on some criterion such as position or appearance. If the tracker is not returning IDs, it may be because the criterion used for association is not strong enough, or there is some other issue with the implementation. One possible solution could be to try adjusting the parameters of the tracker and detector to improve the association.

2. To pass detections to the tracker and update the tracks, you can convert the output of detector.detect() to the format expected by the tracker using a function such as xyxy2xywh(), which converts bounding box coordinates from x1,y1,x2,y2 to x,y,w,h format. You can then pass this to the tracker.update() function, which will update the tracks based on the detections.
3. If you are using the tracker without the detector and are still not getting IDs, it may be that the ID generation code is missing or has not been implemented correctly. Again, you may need to adjust the parameters and/or the implementation to get the desired result.

[vaventt]

@glenn-jocher thanks for the links, I have tried your recommendation, your second option with go clone and next command haven't worked, I'm not a docker professional, but it breaks on the command RUN pip install -e lap/ in Dockerfile, about your first recommendation: when I input into requirements.txt lap==0.4.0 I always got this problem at the stage of building container, not even running it:

#13 82.02           raise ImportError('lap requires numpy, '
#13 82.02       ImportError: lap requires numpy, please "pip install numpy".
#13 82.02       [end of output]
#13 82.02   
#13 82.02   note: This error originates from a subprocess, and is likely not a problem with pip.
#13 82.02 error: legacy-install-failure
#13 82.02 
#13 82.02 × Encountered error while trying to install package.
#13 82.02 ╰─> lap
#13 82.02 
#13 82.02 note: This is an issue with the package mentioned above, not pip.
#13 82.02 hint: See above for output from the failure.

After that I have added numpy before lap==0.4.0 in my requirements.txt, when docker runs it's last command in Dockerfile RUN pip install --upgrade pip && pip install -r requirements.txt and the problem stayed the same, my requirements.txt

numpy
lap==0.4.0
pandas
fastapi
ultralytics
uvicorn[standard]
python-multipart

When I move numpy and lap==0.4.0 to the Dockerfile with commands RUN pip install numpy and RUN pip install lap==0.4.0 I got next error:

#14 10.22       INFO: g++: lap/_lapjv.cpp
#14 10.22       
#14 10.22       
#14 10.22       [Errno 2] No such file or directory: 'g++'
#14 10.22       
#14 10.22       
#14 10.22       INFO: g++: lap/lapmod.cpp
#14 10.22       INFO: g++: lap/lapjv.cpp
#14 10.22       
#14 10.22       
#14 10.22       [Errno 2] No such file or directory: 'g++'
#14 10.22       
#14 10.22       
#14 10.22       
#14 10.22       
#14 10.22       [Errno 2] No such file or directory: 'g++'
#14 10.22       
#14 10.22       
#14 10.22       error: Command "g++ -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/lib/python3.8/site-packages/numpy/core/include -Ilap -I/usr/local/lib/python3.8/site-packages/numpy/core/include -Ibuild/src.linux-x86_64-3.8/numpy/distutils/include -I/usr/local/include/python3.8 -c lap/_lapjv.cpp -o build/temp.linux-x86_64-3.8/lap/_lapjv.o -MMD -MF build/temp.linux-x86_64-3.8/lap/_lapjv.o.d -msse -msse2 -msse3" failed with exit status 127
#14 10.22   note: This error originates from a subprocess, and is likely not a problem with pip.
#14 10.22 error: legacy-install-failure
#14 10.22 
#14 10.22 × Encountered error while trying to install package.
#14 10.22 ╰─> lap

[vaventt]

@Laughing-q magic, I have seen someone with similar problem to mine, he couldn't use botsort algorithm, but with ByteTrack everything worked, nevertheless it's not that important, I'm still happy and grateful for the new argument persist=True. I have a new question, have you tried locally on your computer with yolo8n.pt to run this chunk of code in separate Jupyter cells:

1st cell

im1 = Image.open('traffic.jpg').convert('RGB')
results = model.track(source=im1, persist=True, classes=[2,5,7], tracker='bytetrack.yaml')
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

2nd cell

im2 = Image.open('traffic2.jpg').convert('RGB')
results = model.track(source=im2, persist=True, classes=[2,5,7], tracker='bytetrack.yaml')
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

When I run second cell on new image, on first run on image is like only 1 bbox with id, then when I rerun this cell second time everything become ok, almost all cars that were on the image, was found and assigned with their respective ids

[glenn-jocher]

@SOUMIA21 it's great to hear that the ByteTrack tracker worked for you. Regarding your question, it is expected that when you run the tracker on a new image in a separate cell, the tracker may need one or more frames to detect and assign IDs to all of the objects in the scene. This is because the tracker relies on comparing object positions in successive frames to determine which objects are the same across frames, and it may take a few frames for all objects to be detected and consistently tracked. Additionally, if the image is significantly different from the previous image (e.g., a different scene or camera angle), the tracker may need some additional time to re-establish which objects correspond to which IDs. Overall, the persist=True parameter should help to improve tracking consistency across frames, but it is still important to be aware of potential issues when using the tracker on new frames.

[Laughing-q]

@Laughing-q magic, I have seen someone with similar problem to mine, he couldn't use botsort algorithm, but with ByteTrack everything worked, nevertheless it's not that important, I'm still happy and grateful for the new argument persist=True. I have a new question, have you tried locally on your computer with yolo8n.pt to run this chunk of code in separate Jupyter cells:

1st cell

im1 = Image.open('traffic.jpg').convert('RGB')
results = model.track(source=im1, persist=True, classes=[2,5,7], tracker='bytetrack.yaml')
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

2nd cell

im2 = Image.open('traffic2.jpg').convert('RGB')
results = model.track(source=im2, persist=True, classes=[2,5,7], tracker='bytetrack.yaml')
res_plotted = results[0].plot()
Image.fromarray(res_plotted)

When I run second cell on new image, on first run on image is like only 1 bbox with id, then when I rerun this cell second time everything become ok, almost all cars that were on the image, was found and assigned with their respective ids

yes that's how bytetrack works, first time the tracker sees the objects it would mark it as a unconfirmed state and objects will be assigned if it still matches in the next frame then marked as confirmed state. you might be interested in the whole pipeline in this comment. https://github.com/ultralytics/ultralytics/pull/1890#issuecomment-1500884646

[ChaoYuanNMFC]

I have the same problem with package "lap", when using customized segment model to track objects. However, the model works well in the "yolo predict". Even if the error occurred as "ModuleNotFoundError: No module named 'lap'", when I checked its installation, it suggested "Requirement already satisfied: lap in c:\users\legion\miniconda3\envs\yolo-seg\lib\site-packages (0.4.0)". The environment is win 10 + miniconda + python 3.7. could you tell why and how to solve this.

[glenn-jocher]

@ChaoYuanNMFC it appears that you are receiving a ModuleNotFoundError when trying to use the lap package in a custom segmentation model for object tracking. Despite having installed the package, it seems that the necessary module is not being found when you run the code. This issue may be caused by the package not being properly installed or not being added to the virtual environment where you are running the code.

To solve this issue, you can try the following steps:

1. Check if the lap package is installed in the correct environment where you are running the code. You can verify this by running pip list command in your command prompt or terminal and searching for the lap package in the list of installed packages.
2. Make sure that the virtual environment where you are running the code is activated and that the lap package has been added to the environment. You can activate the environment by running conda activate environment_name command in your terminal or command prompt, where environment_name is the name of your virtual environment.
3. If the package is not installed or not added to the environment, you can try installing it again using pip install lap command.
4. If the issue persists, you can try reinstalling the entire virtual environment and installing all necessary packages including lap.

It is also possible that there is an issue with the code itself or with the way it is being run, so it may be helpful to review the code and ensure that it is consistent with the documentation for the lap package. Overall, the issue appears to be related to the package installation and environment setup, and should be solvable with the steps listed above.

[Chirawat]

Hello! Yes, you can use the YOLOv8 Builtin Tracker for multi-object tracking on video frames read by OpenCV. The tracker can be initialized on a single frame and then updated on subsequent frames.

Here is a brief overview of how you can do it:

1. Initialize the detector and the tracker
2. Read a frame from OpenCV using cap.read()
3. Pass the frame through the detector and get the detections
4. Pass the detections to the tracker and update the tracks
5. Repeat steps 2-4 for each frame

Here is some sample code to get you started:

import cv2
from yolov5 import Detector
from yolov5.utils.bbox import xyxy2xywh
from yolov5.utils.tracker import Tracker

# Initialize the detector and the tracker
detector = Detector(weights='yolov5s.pt')
tracker = Tracker(threshold=0.5)

# Open the video capture
cap = cv2.VideoCapture('test.mp4')
while cap.isOpened():
    # Read a frame
    ret, frame = cap.read()
    if not ret:
        break

    # Pass the frame through the detector and get the detections
    detections = detector.detect(frame)

    # Pass the detections to the tracker and update the tracks
    tracker.update(xyxy2xywh(detections))

    # Draw the tracks on the frame
    for track in tracker.tracks:
        cv2.rectangle

I have the same question with @mohamedamine99, could you give us example code of YOLOv8 instead of YOLOv5 with the same idea?

I have tried many ways but still stuck with the code.

1. persist=True is not useable with Ultralytics version 8.0.59. error message is SyntaxError: 'persist' is not a valid YOLO argument.
2. I set stream=True of using model.track() agrument and process the results outside it. Error message is TypeError: 'generator' object is not subscriptable

When I set the argrument source=... and show=True of model.track() everything works fine but I want to add more information to a frame. Thank You in Advance

[glenn-jocher]

@Chirawat certainly! The approach for using YOLOv8 as a tracker is very similar to that of YOLOv5. You can initialize the detector and the tracker, read a video frame, pass it through the detector to get the detections, and then pass these detections to the tracker to update the tracks.

One difference between YOLOv5 and YOLOv8 is that persist=True is not a valid YOLOv8 argument, so you'll need to find another way to persist the tracker. One option is to use the stream=True argument in model.track() to generate results as a generator, and then process these results outside of the model.track() loop to keep track of the detections and IDs over multiple frames.

It's also worth noting that if you set source=... and show=True in model.track(), you won't be able to add additional information to the generated frames. Instead, you can process the frames outside of model.track() and add your desired information before displaying the modified frames.

Overall, the key concepts for using YOLOv8 as a tracker are very similar to those of YOLOv5, and the main differences lie in the specific arguments and syntax used for each version.

[Chirawat]

@Chirawat certainly! The approach for using YOLOv8 as a tracker is very similar to that of YOLOv5. You can initialize the detector and the tracker, read a video frame, pass it through the detector to get the detections, and then pass these detections to the tracker to update the tracks.

One difference between YOLOv5 and YOLOv8 is that persist=True is not a valid YOLOv8 argument, so you'll need to find another way to persist the tracker. One option is to use the stream=True argument in model.track() to generate results as a generator, and then process these results outside of the model.track() loop to keep track of the detections and IDs over multiple frames.

It's also worth noting that if you set source=... and show=True in model.track(), you won't be able to add additional information to the generated frames. Instead, you can process the frames outside of model.track() and add your desired information before displaying the modified frames.

Overall, the key concepts for using YOLOv8 as a tracker are very similar to those of YOLOv5, and the main differences lie in the specific arguments and syntax used for each version.

@glenn-jocher Thank you for quickly response. I solved my problem with the solution above which is set stream=True and process outside of the model.track(). Here is my code to process video for counting motorcycle as education purpose for all.

`import cv2 from ultralytics import YOLO import numpy as np import torch

model = YOLO('yolov8n.pt')

results = model.track(source="20230402125142.MP4", classes=3, stream=True)

for result in results: boxes = result.boxes frame = result.orig_img

for box in boxes:
    id = box.id

    x1, y1, x2, y2 = np.squeeze(box.xyxy.tolist())
    x1 = int(x1)
    y1 = int(y1)
    x2 = int(x2)
    y2 = int(y2)
    cv2.rectangle(frame, (x1,y1), (x2,y2), (0,255,0), thickness=2)

    # Define the text string and its properties
    if id is not None:
        text = str(int(id.item()))
        org = (x1,y1-10)
        fontFace = cv2.FONT_HERSHEY_SIMPLEX
        fontScale = 1
        color = (0, 255, 0)
        thickness = 2
        lineType = cv2.LINE_AA

        # Put the text on the image
        cv2.putText(frame, text, org, fontFace, fontScale, color, thickness, lineType)

cv2.imshow('frame', frame)
cv2.waitKey(1)

`

[tongchangD]

wath does stream mean

[uma-oo]

@tongchangD stream is used specially when we are dealing with videos or real time feeds , when it is set to True, it returns a generator, and if it is set to False then the results of our model will be a list or an array instead. We use the stream in general if we want to change its default value which is False when dealing with videos as i said and that's because generators consume mush less memory than lists and arrays in python

[uma-oo]

@glenn-jocher Hi there i have a question about the tracker, is there any way to make the tracker remember the object we are tracking if we lose the detection of that object for a certain number of frames ? i was using SORT with yolov8 for a project where i'll be always needing the coordinates of the objects for each frame for a specifc task , in SORT tacker there is the parameter max_age so when setting this parameter to 20 for example we can remember the object if lose its detection using the model of yolov8 but i don't want to work with SORT anymore because it has some problems , and it doesn't fit my expectations on how i want my project to be, so i used yolov8 to track the objects with bytetracker, but if the object is not detected by the model i lose its track, how can i make my object tracker remember the objects for a specific number of frames waiting for the object to be detected again ?