openvinotoolkit / training_extensions

Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™
https://openvinotoolkit.github.io/training_extensions/
Apache License 2.0
1.14k stars 443 forks source link

Geti classification labels messed up by OTX #2397

Closed ssainis-0131 closed 1 year ago

ssainis-0131 commented 1 year ago

The labels from the OTX are completely different order than what is coming in from Geti Deployment

Steps to Reproduce

  1. Create a geti classification project with multiple labels (non hierarchical) and train a model
  2. Take an image from the training data and pass it to the model and look a the prediction
  3. The prediction will be wrong.
  4. Walking through the trace of the load_inference method, you will see it is creating an _inference_conveter in the deployment object using the get_labels method on line 340 of otx/api/entities/label_schema.py

line 340 currently reads

labels = {label for group in self._groups for label in group.labels if include_empty or not `label.is_empty}

As the "set" function destroys the local order of the list it is asked to convert into a set, we get label order that is completey different from the input label list.

My proposed fix to his is

labels = [label for group in self._groups for label in group.labels if include_empty or not label.is_empty]
# remove duplicates
labels = [label for i, label in labels if label not in labels[:i]]

This should preserve the order.

I have tested this at my end. I recommend implementing it in the main code.

Environment: Bug is independent of the environment

goodsong81 commented 1 year ago

Thank you for the report! This PR fixes the issue: https://github.com/openvinotoolkit/training_extensions/pull/2369. (also applied to develop branch).