openvinotoolkit / datumaro

Dataset Management Framework, a Python library and a CLI tool to build, analyze and manage Computer Vision datasets.
https://openvinotoolkit.github.io/datumaro/
MIT License
530 stars 130 forks source link

custom importer/extractor not loading? #404

Closed jonnyyu closed 3 years ago

jonnyyu commented 3 years ago

Hi, I saw there's a comment about how to add a custom importer/extractor, but it seems not working.

https://github.com/openvinotoolkit/datumaro/blob/d75c2343b747440cc6703adb7b9a1efc89c4509d/datumaro/cli/contexts/project/__init__.py#L116

I tried putting myimporter.py to .datumaro/importers folder

from datumaro.components.extractor import Importer

class MyImporter(Importer):
    @classmethod
    def find_sources(cls, path):
        return []

then run following code in python interactive under project root.

>>> from datumaro.components.environment import Environment
>>> env = Environment()
>>> print([importer for importer in env.importers])
>>> ['imagenet_txt', 'yolo', 'cityscapes', 'label_me', 'coco_captions', 'coco_image_info', 'coco', 'coco_instances', 'coco_labels', 'coco_panoptic', 
'coco_person_keypoints', 'coco_stuff', 'cifar', 'image_dir', 'wider_face', 'lfw', 'sly_pointcloud', 'market1501', 'image_zip', 'mot_seq', 'kitti_
detection', 'kitti', 'kitti_segmentation', 'kitti_raw', 'cvat', 'mots', 'imagenet', 'open_images', 'mnist_csv', 'datumaro', 'vgg_face2', 'mnist',
 'voc_action', 'voc_classification', 'voc_detection', 'voc', 'voc_layout', 'voc_segmentation', 'icdar_text_localization', 'icdar_text_segmentatio
n', 'icdar_word_recognition', 'camvid']

Then I tried force loading plugin from the specific directory

>>>  env.load_plugins('.datumaro/importers')
>>> print([importer for importer in env.importers])
['imagenet_txt', 'yolo', 'cityscapes', 'label_me', 'coco_captions', 'coco_image_info', 'coco', 'coco_instances', 'coco_labels', 'coco_panoptic', 
'coco_person_keypoints', 'coco_stuff', 'cifar', 'image_dir', 'wider_face', 'lfw', 'sly_pointcloud', 'market1501', 'image_zip', 'mot_seq', 'kitti_
detection', 'kitti', 'kitti_segmentation', 'kitti_raw', 'cvat', 'mots', 'imagenet', 'open_images', 'mnist_csv', 'datumaro', 'vgg_face2', 'mnist',
 'voc_action', 'voc_classification', 'voc_detection', 'voc', 'voc_layout', 'voc_segmentation', 'icdar_text_localization', 'icdar_text_segmentatio
n', 'icdar_word_recognition', 'camvid', 'my']
zhiltsov-max commented 3 years ago

Hi, it looks like the plugin directory was overlooked in the last implementation update. It should be called here: https://github.com/openvinotoolkit/datumaro/blob/develop/datumaro/components/project.py#L417

<project_dir>/.datumaro/extractors should be <project_dir>/.datumaro/plugins now.

Thank you for reporting the issue!