search

voxel51 / fiftyone

Refine high-quality datasets and visual AI models
https://fiftyone.ai
Apache License 2.0
8.84k stars 557 forks source link

Add a label field for captions #291

Open brimoor opened 4 years ago

brimoor commented 4 years ago

A common use case with multimodal models is having extended text captions to go with images. For example, the dataset may be curated by scraping a google images search and storing (caption, title of page) or (caption, first sentence of page) pairs.

FityOne could naturally support these dataset types by providing a Caption(Label) field that stores an extended text string. The reason for the dedicated Label subclass would be to indicate to the App that such labels should be rendered as longer text descriptions on the image, not in the label "chin" below the image.

For discussion.

benjaminpkane commented 3 years ago

When added, a fiftyone example notebook interacting with OpenAI's CLIP would be a great addition.

Their own notebook currently uses matplotlib.