With this refactor, VoxelGPT is officially a chatbot. This PR adds chat-style responses for queries related to the FiftyOne docs, and computer vision more broadly.
How?
swaps out validate_query() for classify_query_intent(), which classifies the query as
display: create a DatasetView
documentation: answer based on the FiftyOne docs
computer vision: a broader computer vision query
invalid: anything else
adds computer_vision_query_dispatcher link, which implements run_computer_vision_query(), which is effectively a light wrapper around ChatGPT itself, without any additional steps.
adds docs_query_dispatcher link, which implements run_docs_query(). This spins up a Chroma database from precomputed embeddings for all of the docs - see _generate_embeddings() and _create_vectorstore() - and uses this vector db for RetrievalQA to answer based on relevant docs.
Notes:
Needed to reach inside the Chroma LangChain class to set the contents with pre-computed embeddings so we don't need to compute them from scratch every time.
This Chroma DB is persisted so it only task a few seconds to load, as opposed to initializing from scratch (~10-20 seconds to respond, as opposed to 4 minutes without this). Would love to optimize this even further!
Query intent classification is very good, which is nice because with the examples in fiftyone_query_intent_examples.csv, we are bumping up against the model's context limit.
General computer vision queries take the longest, which is odd. Currently they take 20-30s, so maybe I'm doing something wrong?
Examples:
computer vision: How do I train a model?
Training a model involves several steps, including data preparation, model selection, hyperparameter tuning, and evaluation. Here is a general overview of the process:
Data preparation: Collect and preprocess data to ensure it is in a format that can be used for training. This may involve cleaning, normalization, and feature engineering.
Model selection: Choose an appropriate model architecture based on the problem you are trying to solve. This may involve selecting from a range of pre-existing models or building a custom model.
Hyperparameter tuning: Adjust the model's hyperparameters to optimize its performance. This may involve using techniques such as grid search or random search.
Training: Train the model on the prepared data using an appropriate algorithm. This may involve using techniques such as stochastic gradient descent or backpropagation.
Evaluation: Evaluate the model's performance on a separate test set to ensure it is generalizing well to new data.
There are many resources available to help with each of these steps, including online courses, textbooks, and research papers. It's important to keep in mind that training a model can be a complex and iterative process, and it may take several rounds of experimentation to achieve the desired results.
documentation: How can I load a dataset into FiftyOne?
You can load a dataset into FiftyOne by using the from_dir() method of the Dataset class. You need to specify the directory containing the dataset, the type of the dataset being imported, and a name for the dataset. If your data is stored in one of the many common formats supported natively by FiftyOne, you can use the following simple pattern:
import fiftyone as fo
# A name for the dataset
name = "my-dataset"
# The directory containing the dataset to import
dataset_dir = "[/path/to/dataset](https://file+.vscode-resource.vscode-cdn.net/path/to/dataset)"
# The type of the dataset being imported
dataset_type = fo.types.COCODetectionDataset # for example
dataset = fo.Dataset.from_dir(
dataset_dir=dataset_dir,
dataset_type=dataset_type,
name=name,
)
If your data is in a custom format, you can write a simple loop to load your data into FiftyOne.
display: Show me 10 random images from the dataset
With this refactor, VoxelGPT is officially a chatbot. This PR adds chat-style responses for queries related to the FiftyOne docs, and computer vision more broadly.
How?
validate_query()forclassify_query_intent(), which classifies the query asdisplay: create aDatasetViewdocumentation: answer based on the FiftyOne docscomputer vision: a broader computer vision queryinvalid: anything elsecomputer_vision_query_dispatcherlink, which implementsrun_computer_vision_query(), which is effectively a light wrapper around ChatGPT itself, without any additional steps.docs_query_dispatcherlink, which implementsrun_docs_query(). This spins up a Chroma database from precomputed embeddings for all of the docs - see_generate_embeddings()and_create_vectorstore()- and uses this vector db forRetrievalQAto answer based on relevant docs.Notes:
ChromaLangChain class to set the contents with pre-computed embeddings so we don't need to compute them from scratch every time.fiftyone_query_intent_examples.csv, we are bumping up against the model's context limit.Examples:
computer vision: How do I train a model?documentation: How can I load a dataset into FiftyOne?display: Show me 10 random images from the dataset