openvinotoolkit / training_extensions

Train, Evaluate, Optimize, Deploy Computer Vision Models via OpenVINO™
https://openvinotoolkit.github.io/training_extensions/
Apache License 2.0
1.14k stars 443 forks source link

training accuracy reported as 0.0 with CUB hierarchical label for the classification #2579

Closed yunchu closed 1 year ago

yunchu commented 1 year ago

Describe the bug Training accuracy reported as 0.0 with CUB hierarchical label for the classification here is the part of regression_config.json for this testing image

Steps to Reproduce

  1. run classification regression TC
    $ CI_DATA_ROOT=/home/yunchule/data/ci_datasets/ tox -vvv -e tests-cls-py310-pt1 -- tests/regression/classification/test_classification.py::TestRegressionHierarchicalLabelClassification::test_otx_train[Custom_Image_Classification_EfficientNet-V2-S]
  2. check the log
    >       assert test_result["passed"] is True, test_result["log"]
    E       AssertionError: [EfficientNet-V2-S] Performance: (0.0) < Criteria: (0.5418), threshold: 0.1.
    E       assert False is True

Environment:

wonjuleee commented 1 year ago

The problem was because of the hierarchical classification dataset generation from the original CUB data. Since CUB categories are composed of proper noun of birds, it is not appropriate to split the category names into hierarchies.

yunchu commented 1 year ago

Thanks for your update @wonjuleee I will close this issue.

goodsong81 commented 1 year ago

Found root cause, and fixed in https://github.com/openvinotoolkit/training_extensions/pull/2626