Prototype YOLO training pipeline using Florence-2 as dataset generator

[sah4jpatel]

Florence-2 is a new VLM that is capable of very accurately finding specific objects within images such as "purple balloon," "red blimp," "yellow square" even in terrible conditions such as low light, poor angle, and incomplete or partial object presence. Using this tool in real-time is likely not possible yet given that the ONNX support is still a bit too unrefined to work with RKNN. Even if RKNN worked, this model likely couldn't run in real-time. YOLO V8 or V10 might be optimal models that can be trained on this generated dataset.

Proposed plan:

graph LR;
    A["Split Competition Dataset\nFootage into Frames"]-->B["Process Images with Florence-2\n with OPEN_VOCABULARY_DETECTION tag"];
    B-->C["Feed Image-BBox dataset into YOLO"];

[sah4jpatel]

thoughts? @Rhyme0730

[Rhyme0730]

What's the meaning of 'Even if RKNN worked, this model likely couldn't run in real-time' ? So you mean we can use this VLM to train our model offline in a better way, and Yolov8 or Yolov10 can be compatible with this VLM model. Is that right?

[sah4jpatel]

Yea we can just use a VLM as a way of generating our dataset which can be used to train a YOLO model.

I can build out that dataset generation script once I get the footage we recorded from last competition. Even the smallest VLMs I've seen can't run on SBCs in a way where the inference time is < 1 second even.

[Rhyme0730]

I see, so the trained yolo model can be effective even in the terrible conditions like strong light. It's worth a try and Opi is good enough to run a yolov8/v10 model. Btw do you have past dataset now? I also want to get them to train a yolov5 model and see the performance.

[sah4jpatel]

YOLOv10 on a hand-created small dataset works well enough. way be able to close this issue in favor of a new issue #35

[sah4jpatel]

Florence is cool but not great at bounding box work yet, probably needs a better prompt and supporting software