Scope out tasks for API & Software Use Direction

ManifoldRG / AgentForge

The AgentForge project focuses on building general tooling to construct multicapability AI systems by composing skills and models together, and having intelligent processes use those skills/models.

GNU General Public License v3.0

17 stars 0 forks source link

Scope out tasks for API & Software Use Direction #35

Open harshsikka opened 9 months ago

harshsikka commented 9 months ago

Context: Somewhat dependent on #3, though we have some clarity on what tasks to break out in this research direction. The following should be issues

[ ] Review the current API datasets (from https://gorilla.cs.berkeley.edu/ and other works), and start building a living dataset and benchmark for API calling.
[ ] Review SoTA function calling in LLMs
[ ] Rank/Review the most useful APIs for knowledge work and beyond
[ ] Determine the GUI objective
[ ] Determine whether an agent architecture or an end to end model is the right way forward (dependent on #3 )
[ ] view DuckTrack and OpenAdapt to understand if they are the right tools for dataset creation for GUI datasets
[ ] initiate collaborations w/ Gorilla, DuckAI, etc

Output: break the above into issues with leads

pranavguru commented 9 months ago

Strategies to guide API survey:

Survey the kind of APIs covered in existing API and function-calling datasets
Analyze the distribution of APIs over the above-mentioned datasets
Evaluate if there is a way to find the most frequently used APIs, and whether they can be incorporated in the dataset to be created
Think about how a dataset can be created based on findings from the above 3 points, which will enable the training of a model with a large number of capabilities

harshsikka commented 9 months ago

following on our discussion, some links to check out:

pranavguru commented 9 months ago

Another resource - https://huggingface.co/datasets/liminghao1630/API-Bank

harshsikka commented 9 months ago

@pranavguru API/Function Calling Dataset Analysis

Domain + Coverage
example - how is the dataset structured
what is SoTA in function calling

pranavguru commented 7 months ago

Another relevant resource for this - https://restgpt.github.io/