YOLO: Combined model is needed

[dmpetrov]

Description

To vissualize all possible signals we need something like this:

class MyYolo(DataModel):   # <--- this or something similar should be built-in, ideally with a single function like `from_result()`
    bbox: model.ultralytics.YoloBBoxes
    segm: model.ultralytics.YoloSegments
    poses: model.ultralytics.YoloPoses

def process(yolo: YOLO, yolo_segm: YOLO, yolo_pose: YOLO, file: File) -> MyYolo:
    img = Image.open(BytesIO(file.read()))
    return MyYolo(
        bbox=model.ultralytics.YoloBBoxes.from_results(yolo(img, verbose=False)),
        segm=model.ultralytics.YoloSegments.from_results(yolo_segm(img, verbose=False)),
        poses=model.ultralytics.YoloPoses.from_results(yolo_pose(img, verbose=False))
    )

(
    DataChain
    .from_storage("gs://mpii-human-pose")
    .limit(100)
    .setup(
        yolo=lambda: YOLO("yolo11n.pt"),
        yolo_segm=lambda: YOLO("yolo11n-seg.pt"),
        yolo_pose=lambda: YOLO("yolo11n-pose.pt")
    )
    .map(my_yolo=process)
    .save("mpii-human-pose-segment")
)

An idea: from_result() can have options like from_result(bbox=True, segm=True, poses=True)

[shcheklein]

To vissualize all possible signals we need something like this:

@dmpetrov is it opposed to something that we have now? can you give a bit more details please?

[dmpetrov]

@shcheklein could you please clarify the question?

[shcheklein]

@dmpetrov I guess my question is - that code that you shared, is it something that we do now? E.g. do I have to do a custom MyYolo to visualize multiple signals now?

[dreadatour]

E.g. do I have to do a custom MyYolo to visualize multiple signals now?

To visualize multiple kind of signals (e.g. both poses and segments the same time) — yes, either custom model or two signals with build-in models. This will also requires setup multiple models (like one for poses and another one for segments).

[shcheklein]

yes, either custom model or two signals with build-in models.

yep, thanks! Q: what is the value then to do a combined model, can we just always return multiple independent (built in?) models?

[dreadatour]

Since we now have toolkit module, I can add helpers there for YOLO visualization 👀