I need some tips for creating a new iterator in C++

[ascust]

Hi @zhreshold , I recently intend to implement a new iterator in C++ and I saw you have created one on your own. Would you give me some tips, like the general procedures. What classes should be created and modified etc. I have had a look at your code, but it seems multiple classes are involved, which looks a bit confusing. Also I want to load data for segmentation, so the data format might be different from classification or detection ones. Any suggestions. I would really appreciate it.

[zhreshold]

Well, if you are trying to write a iterator for segmentation, then all these classes I wrote will be necessarily rewritten to have full functionalities.

io/iter_image_det_recordio.cc: ImageDetLabelMap: used to parse labels ImageDetRecParserParam: parse parameters such as path, data_shape, threads, etc... ImageDetRecordIOParser: implement Init() to open the record file, impl ParseNext() to read next image/label, including call augmenter. ImageDetRecordParam/ImageDetRecordIter : wrapper to return proper format for the data/label

io/image_det_aug_default.cc DefaultImageDetAugmentParam: parse parameters for the augmentations class DefaultImageDetAugmenter : public ImageAugmenter : implement the Process() to do the real stuff for augmentation.

When you register the Iter like this:

MXNET_REGISTER_IO_ITER(ImageDetRecordIter)
.describe("Create iterator for image detection dataset packed in recordio.")
.add_arguments(ImageDetRecParserParam::__FIELDS__())
.add_arguments(ImageDetRecordParam::__FIELDS__())
.add_arguments(BatchParam::__FIELDS__())
.add_arguments(PrefetcherParam::__FIELDS__())
.add_arguments(ListDefaultDetAugParams())
.add_arguments(ImageDetNormalizeParam::__FIELDS__())
.set_body([]() {
  return new PrefetcherIter(
        new BatchLoader(
            new ImageDetNormalizeIter(
                new ImageDetRecordIter<real_t>())));
});

You are registering all the parameters into ImageDetRecordIter, iter = mx.io.ImageDetRecordIter(...) when you call iter.next()->ImageDetRecordIOParser::ParseNext()->Augmenter::Process(), and finally return data/labels.

Hope this helps.