custom object detection/segmentation

[woody0105]

Summary

We want to allow users to create custom model that can track custom objects. User created dataset and trained model will be public and as more users involved, the dataset and models will be plentified.

Tools needed

To be able to allow users to add custom objects to instance segmentation, we should provide:

• data annotation toolkit (desktop or web app to allow users to tag the custom objects)
• training toolkit

Considerations

• maximum number of classes that can be detected/tracked
• whether accuracy will be affected as the number of classes increase

[woody0105]

Instance segmentation steps in general

1. Prepare labeled image dataset using labelme. Image annotations should be polygon.
2. (Optional depending on segmentation model) Convert annotation data to be compatible with the model
3. Train the model

Dataset collection/preparation

• Opensource image labeling tools and outsource to community
• How to validate of annotation quality: undecided

Training and model sharing

• Users self train models and share trained models along with the dataset that is used to train the model
• Livepeer to take requests to train custom models (since it requires access to GPU hardware and some expertise in ML)
• Outsource custom training jobs to community??

Yolact (Current implementation)

Yolov7 (Released in July, 2022, fastest & most accurate)