voxel51 / voxelgpt

AI assistant that can query visual datasets, search the FiftyOne docs, and answer general computer vision questions
https://gpt.fiftyone.ai
Apache License 2.0
234 stars 17 forks source link

[FEATURE] Handle unmatched class names with similarity search #38

Closed jacobmarks closed 1 year ago

jacobmarks commented 1 year ago

The underlying GPT-3.5 model was trained on the internet, so it has a tendency to use match() and filter_labels() view stages even when the label class does not actually exist in the dataset. This patches this by

Example scenario where this would fix the problem:

import fiftyone as fo
import fiftyone.brain as fob
import fiftyone.zoo as foz

from gpt_view_generator import ask_gpt

dataset = foz.load_zoo_dataset("quickstart")
fob.compute_similarity(
   dataset,
   model='clip-vit-base32-torch`,
   brain_key='clip'
)

view = ask_gpt(dataset, query = "images of elks")

Will result in ... Okay, I'm going to load dataset.sort_by_similarity('elk', brain_key = 'clip', k = 25)