[BUG] ImageSegmentationDirectory export fails with OpenCV zero area size error

samhodge-aiml commented 2 years ago

Instructions

Thank you for submitting an issue. Please refer to our issue policy for information on what types of issues we address.

Please fill in this template to ensure a timely and thorough response.

Place an "x" between the brackets next to an option if it applies. Example:
- [x] Selected option
Please delete this section (all content above this line) before submitting the issue

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 20.04 LTS
FiftyOne installed from (pip or source): yes| conda install --prefix ./env python=3.9 && pip3 install fiftyone pycocotools
FiftyOne version (run fiftyone --version): FiftyOne v0.16.6, Voxel51, Inc.
Python version: Python 3.9.13

Commands to reproduce

As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue. Application steps can be described in the next section.

python3 ./make_data.py

make_data.py

#!/usr/bin/env python3
import fiftyone as fo
import fiftyone.zoo as foz
from pathlib import Path

FOLDER_STORE = "coco"
FOLDER = Path(__file__).parent / FOLDER_STORE
FOLDER.mkdir(parents=True, exist_ok=True)
OUTPUT_FOLDER = Path(__file__).parent / "dataset1"
OUTPUT_FOLDER.mkdir(parents=True, exist_ok=True)
dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="train",
    classes=["person"],
    label_types=["segmentations"],
    shuffle=True,
    max_samples=8000,
    dataset_dir=FOLDER
)

dataset_type = fo.types.ImageSegmentationDirectory
label_field = "ground_truth"  

dataset.export(
    export_dir=f"{OUTPUT_FOLDER.as_posix()}",
    dataset_type=dataset_type,
    label_field=label_field,
)

Describe the problem

OpenCV error occurs

(/media/sam/aimlwork/coco-easy/env) sam@rotobot-ampere-dev:/media/sam/aimlwork/coco-easy$ python3 make_data.py 
Downloading split 'train' to '/media/sam/aimlwork/coco-easy/coco/train' if necessary
Found annotations at '/media/sam/aimlwork/coco-easy/coco/raw/instances_train2017.json'
Sufficient images already downloaded
Existing download of split 'train' is sufficient
Loading 'coco-2017' split 'train'
 100% |███████████████████████████████████████████████████████████████| 8000/8000 [1.7m elapsed, 0s remaining, 92.7 samples/s]      
Dataset 'coco-2017-train-8000' created
Directory '/media/sam/aimlwork/coco-easy/dataset1' already exists; export will be merged with existing files
  31% |███████████████████--------------------------------------------| 2519/8000 [25.0s elapsed, 51.1s remaining, 124.4 samples/s] 
Traceback (most recent call last):
  File "/media/sam/aimlwork/coco-easy/make_data.py", line 24, in <module>
    dataset.export(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/core/collections.py", line 6717, in export
    _export(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/core/collections.py", line 8804, in _export
    foud.export_samples(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/utils/data/exporters.py", line 339, in export_samples
    write_dataset(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/utils/data/exporters.py", line 392, in write_dataset
    _write_image_dataset(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/utils/data/exporters.py", line 825, in _write_image_dataset
    dataset_exporter.export_sample(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/utils/data/exporters.py", line 2757, in export_sample
    segmentation = label.to_segmentation(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/core/labels.py", line 588, in to_segmentation
    _render_instance(mask, detection, target)
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/fiftyone/core/labels.py", line 1223, in _render_instance
    obj_mask, offset = etai.render_instance_mask(
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/eta/core/image.py", line 967, in render_instance_mask
    mask = resize(mask, width=width, height=height)
  File "/media/sam/aimlwork/coco-easy/env/lib/python3.9/site-packages/eta/core/image.py", line 714, in resize
    return cv2.resize(img, (width, height), *args, **kwargs)
cv2.error: OpenCV(4.6.0) /io/opencv/modules/imgproc/src/resize.cpp:4052: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

Code to reproduce issue

As above

As thoroughly as possible, please provide the Python and/or shell commands used to encounter the issue. Application steps can be described in the next section.

python3 ./make_data.py

make_data.py

#!/usr/bin/env python3
import fiftyone as fo
import fiftyone.zoo as foz
from pathlib import Path

FOLDER_STORE = "coco"
FOLDER = Path(__file__).parent / FOLDER_STORE
FOLDER.mkdir(parents=True, exist_ok=True)
OUTPUT_FOLDER = Path(__file__).parent / "dataset1"
OUTPUT_FOLDER.mkdir(parents=True, exist_ok=True)
dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="train",
    classes=["person"],
    label_types=["segmentations"],
    shuffle=True,
    max_samples=8000,
    dataset_dir=FOLDER
)

dataset_type = fo.types.ImageSegmentationDirectory
label_field = "ground_truth"  

dataset.export(
    export_dir=f"{OUTPUT_FOLDER.as_posix()}",
    dataset_type=dataset_type,
    label_field=label_field,
)

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Please do not use screenshots for sharing text. Code snippets should be used instead when providing tracebacks, logs, etc.

What areas of FiftyOne does this bug affect?

[ ] App: FiftyOne application issue
[ * ] Core: Core fiftyone Python library issue
[ ] Server: Fiftyone server issue

Willingness to contribute

The FiftyOne Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the FiftyOne codebase?

[ * ] Yes. I can contribute a fix for this bug independently.
[ ] Yes. I would be willing to contribute a fix for this bug with guidance from the FiftyOne community.
[ ] No. I cannot contribute a bug fix at this time.

Fix seems easy enough, just test to see if mask has non zero sizes before making the call, else return a blank image.

samhodge-aiml commented 2 years ago

Inserting

    height, width = mask.shape[:2]
    if height == 0:
        return
    if width == 0:
        return

fiftyone/core/labels.py

at line 1223

fixes the issue.

samhodge-aiml commented 2 years ago

Once the fix is applied there are ground truth labels that are not person

see images attached

000000016238

brimoor commented 2 years ago

Hi @samhodge-aiml 👋

Good catch on the error, looks like there needs to be some error handling inserted when an object has a mask with zero pixels.

The second part is expected behavior. By default classes=["person"] means to only load images that contain at least one person, but to load all boxes that exist in those images. If you really only want the person boxes, you can include the optional only_matching=True parameter documented here.

Alternatively, you can use filter_labels() to export only the person boxes from the full dataset.

import fiftyone as fo
import fiftyone.zoo as foz
from fiftyone import ViewField as F

#
# Option 1: only load `person` boxes initially
#

dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="train",
    classes=["person"],
    label_types=["segmentations"],
    only_matching=True,
    shuffle=True,
    max_samples=8000,
)

print(dataset.count_values("ground_truth.detections.label"))
# {'person': 33416}

dataset.export(...)

#
# Option 2: load all labels initially, but filter for the export
#

dataset = foz.load_zoo_dataset(
    "coco-2017",
    split="train",
    classes=["person"],
    label_types=["segmentations"],
    shuffle=True,
    max_samples=8000,
)

print(dataset.count_values("ground_truth.detections.label"))
# {'apple': 194, 'toaster': 6, ..., 'person': 33416, ...}

view = dataset.filter_labels("ground_truth", F("label") == "person")

print(view.count_values("ground_truth.detections.label"))
# {'person': 33416}

view.export(...)

kognat-docs commented 2 years ago

Very cool, thanks for the tutorial. I have to switch context to another project but can get back to this in either 12 or 24 hours.

Thanks a million 🎉

brimoor commented 2 years ago

This is fixed by https://github.com/voxel51/eta/pull/569 and will be resolved in fiftyone>=0.17.

kognat-docs commented 2 years ago

Thanks

voxel51 / fiftyone