Bug: Abnormal Confidence Scores in YOLO-World Inference Due to New Version Bug

[pg56714]

Search before asking

• [X] I have searched the Inference issues and found no similar bug report.

Bug

Sample https://github.com/pg56714/YoloWorldDemo

person wearing green clothes

inference-gpu[yolo-world]==0.13.0 and inference[yolo-world]==0.13.0 Confidence Threshold 0.2 is not ok Confidence Threshold 0.15

Confidence Threshold 0.2 --> It should be the same as version 0.2 of 0.9.15.

inference-gpu[yolo-world]==0.9.15 and inference[yolo-world]==0.9.15 is ok Confidence Threshold 0.15

Confidence Threshold 0.2

Environment

python=3.10 inference==0.13.0 and inference==0.14.0

Minimal Reproducible Example

Same as above

Additional

No response

Are you willing to submit a PR?

• [ ] Yes I'd like to help by submitting a PR!

[PawelPeczek-Roboflow]

Hello there :)

So, to sum up - it seems like:

• the same model version
• the same confidence behaves differently under different inference package version. Were all examples that provided tested in the same moment?

[PawelPeczek-Roboflow]

I remember one switch in YoloWorld weights after initial appearance in inference - if image depicting version 0.9.15 is old - maybe that's the case

[pg56714]

@PawelPeczek-Roboflow

Hello, thank you for your reply.

• the same model version
• the same confidence behaves differently under different inference package versions. Were all examples that provided tested at the same moment?

--> Yes, all tests were conducted on the same day.

• I first tested version 0.9.15, then reinstalled a new environment, and encountered this issue.
• Conversely, the same issue occurred when I first installed a new environment and then tested version 0.9.15.

I remember one switch in YoloWorld weights after initial appearance in inference - if the image depicting version 0.9.15 is old - maybe that's the case.

--> I apologize, but that's not the reason. Firstly, 0.9.15 used the old model, yolo_world/l, but the new version has this issue with all new models.

The old version only supports: YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/l")

https://github.com/roboflow/inference/pull/322 The new version supports: YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/s") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/m") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/l") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/x")

YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-s") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-m") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-l") YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-x")

However, using the YoloWorld code directly from their GitHub does not result in this problem, regardless of whether it is with the new model or the old model.

[pg56714]

Using the scenario described above:

1. With the new versions and the image provided, inputting "person wearing green clothes," none of the models could detect objects meeting the confidence threshold:

   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/s")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/m")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/l")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/x")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-s")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-m")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-l")
   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-x")

2. In version 0.9.15 with the same image, inputting "person wearing green clothes," the model could correctly detect objects meeting the confidence threshold:

   • YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/l")

[pg56714]

You can try it in my repository at https://github.com/pg56714/YoloWorldDemo. Thank.

[pg56714]

YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/l")

inference-gpu[yolo-world]==0.13.0 and inference[yolo-world]==0.13.0 Confidence Threshold 0.2 is not ok Confidence Threshold 0.2 --> It should be the same as version 0.2 of 0.9.15.

Confidence Threshold 0.15

[pg56714]

YOLO_WORLD_MODEL = YOLOWorld(model_id="yolo_world/v2-l")

[pg56714]

@PawelPeczek-Roboflow When only two objects remain in the final situation, if the confidence score condition is met by only one object, a bug occurs.

However, the official GitHub repository displays correctly. It may require you to check if there's an error in the .infer() method. Your assistance is much appreciated, thank you.

results = YOLO_WORLD_MODEL.infer(
    input_image, confidence=confidence_threshold, nms=iou_threshold
)

[PawelPeczek-Roboflow]

ok, thanks for details - will take a look

[PawelPeczek-Roboflow]

ok, found out where the problem is - we have faulty squeeze() in NMS function only revealing bug for singular detection prior to NMS. I expect that to be problematic for other models also - we'll see. np_conf_mask = (np_image_pred[:, 4] >= conf_thresh).squeeze()

[PawelPeczek-Roboflow]

and after 0.9.15 we added post-processing with NMS to yolo-world - which explains why old version is not affected

[PawelPeczek-Roboflow]

created bug-fix PR: https://github.com/roboflow/inference/pull/535 we need to run all our tests to make sure we are not breaking anything else with this bug-fix, so probably will take a while until release - but thanks for the report - that was a very nasty bug!

[pg56714]

Thank you for the fix. However, I would like to wait until the new version has been officially deployed and tested before closing this issue. Could you please inform me which version will include this fix and the expected release date?

Thank you very much for your assistance.

[grzegorz-roboflow]

Hi @pg56714 , we are about to release inference v.0.15.0

[pg56714]

OK. Thank.

[pg56714]

@PawelPeczek-Roboflow @grzegorz-roboflow

new bug #539

[grzegorz-roboflow]

@pg56714 thank you for reporting the other bug! Was the original problem reported in this report resolved?

[pg56714]

I initially thought there was an installation issue, so I didn't test it. I will try it now, thank you. I'll use an older version of gradio first, as the new version has conflicts, which has been noted in #539 .

[pg56714]

Done.