Enhance DatasetItem annotations for semantic segmentation model training use case

openvinotoolkit / datumaro

Dataset Management Framework, a Python library and a CLI tool to build, analyze and manage Computer Vision datasets.

MIT License

551 stars 135 forks source link

Summary

Ticket no. 141410
Datumaro has been maintaining every mask as a binary mask representation. This is because of flexibility on manipulating dataset level management (e.g., label remapping). However, this design can bring performance degradation in case of model training use case. For example, semantic segmentation model training uses 2d mask filled in an integer value for each pixel. However, during importing a dataset, Datumaro should convert it to a collection of binary masks. It makes impossible to use the integer mask directly, so that it requires a 2d complexity computation cost to merge the imported binary masks into an integer mask.
To resolve this performance degradation, this PR introduces a new interface on dataset_item.annotations by introducing Annotation(list[Annotation]) class. This class extends the pure Python list to equip more utility functions. At the same time, the utility function we introduce this time is dataset_item.annotations.get_semantic_seg_mask(). This function bypasses binary mask conversion to construct an integer mask used for semantic segmentation model training.

How to test

Added unit tests to cover this change (mainly in tests/unit/test_annotation.py).
This is a performance test on the semantic segmentation model training use case. Throughput improvement on getting a semantic segmentation mask is ~3x (31.38 it/s -> 99.31 it/s).

Checklist

[x] I have added unit tests to cover my changes.
[ ] I have added integration tests to cover my changes.
[x] I have added the description of my changes into CHANGELOG.
[ ] I have updated the documentation accordingly

License

[x] I submit my code changes under the same MIT License that covers the project. Feel free to contact the maintainers if that's a concern.
[x] I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
#
# SPDX-License-Identifier: MIT

Codecov Report

Attention: Patch coverage is 79.31034% with 6 lines in your changes are missing coverage. Please review.

Please upload report for BASE (releases/1.7.0@3501e97). Learn more about missing BASE report.

Files	Patch %	Lines
src/datumaro/components/annotation.py	76.92%	4 Missing and 2 partials :warning:

Files

Patch %

Lines

src/datumaro/components/annotation.py

76.92%

4 Missing and 2 partials :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## releases/1.7.0 #1503 +/- ## ================================================= Coverage ? 80.78% ================================================= Files ? 276 Lines ? 31511 Branches ? 6356 ================================================= Hits ? 25456 Misses ? 4643 Partials ? 1412 ``` | [Flag](https://app.codecov.io/gh/openvinotoolkit/datumaro/pull/1503/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openvinotoolkit) | Coverage Δ | | |---|---|---| | [ubuntu-20.04_Python-3.10](https://app.codecov.io/gh/openvinotoolkit/datumaro/pull/1503/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openvinotoolkit) | `80.77% <79.31%> (?)` | | | [windows-2022_Python-3.10](https://app.codecov.io/gh/openvinotoolkit/datumaro/pull/1503/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openvinotoolkit) | `80.75% <79.31%> (?)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=openvinotoolkit#carryforward-flags-in-the-pull-request-comment) to find out more.

openvinotoolkit / datumaro