Given time pressure, we aren't going fully with the 2.0 (agentic workflows). However, when this version is complete, it should present some major upgrades.
Capabilities
Can create DatasetViews using almost every view stage in FiftyOne with almost 0% "invalid", as long as you provide the right field names
Generates appropriate filters/view expressions for ints, floats, bools, strings, dates and datetime objects, and lists.
Generation of filters for bounding box area (both rel and abs), and number of detections
Ability to use GeoNear and GeoWithin from natural language descriptions (geocoding)
Can answer questions about your workspace, including datasets you have downloaded, your plugins, and your app config
Can answer questions about your dataset's schema
Decoupling of view creation from view setting in the app — not all dataset queries are asking to set the view.. for instance, aggregations!
Ability to perform aggregations (over fields and expressions) on the entire dataset, or the constructed view
QAWithSources over FiftyOne 0.24.0 docs, all blogs from the Voxel51 Medium publication, and crucial YouTube transcripts from the Voxel51 YouTube Channel
Checklist for v1.5
✅ Add back in chat history
✅ Do data schema inspection given initial view creation plan
✅ Add a replanning stage
✅ Postprocessing of difficult stages, including disambiguation and fallback options
✅ Link for creating view from dataset or starting from current view in the app
✅ Add back in effective query generation from history
✅ Decide what to expose to the user at each step
✅ Revise view stage code blocks shown to user
✅ Redo README
✅ Allow to work with Azure distro of models as well as OpenAI API key
✅ Handle metadata computation automatically if needed
✅ (Moved up) Implement tool use with human approval
Add tool to let the workspace inspection agent inspect the READMEs for plugins, so it has a better sense for how to install and use them
Instead of a static help message, give VoxelGPT some introspection and dynamic help capabilities by routing the help queries to a chain where the context includes info on VoxelGPT's capabilities - what types of queries it can answer, what tools it has access to, etc...
In Store for v2.0
Add Community Slack messages for QA Chain
Give access to core plugins
Handling of important special cases where the syntax is different from usual: {ToPatches, ToEvaluationPatches} followed by {SortBy, GroupBy, FilterField}
Make VoxelGPT pluggable (any operator with a voxelgpt() method, including description and annotations, could be dynamically converted into a tool
Incorporate all of the capabilities implemented so far into a higher level agent framework where VoxelGPT can provide insights into your data
Better handling for edge cases around 3D and grouped data
Add docs query caching for improved response on often-asked questions
Connect to GPT-4 visual interface to answer questions about selected images
Given time pressure, we aren't going fully with the 2.0 (agentic workflows). However, when this version is complete, it should present some major upgrades.
Capabilities
DatasetViews using almost every view stage in FiftyOne with almost 0% "invalid", as long as you provide the right field names0.24.0docs, all blogs from the Voxel51 Medium publication, and crucial YouTube transcripts from the Voxel51 YouTube ChannelChecklist for v1.5
In Store for v2.0
ToPatches,ToEvaluationPatches} followed by {SortBy,GroupBy,FilterField}