Add Multi-Vendor Integration - Serge Priority

[kagrith]

Duplicates

• [X] I have searched the existing issues

Summary 💡

Hopefully i havent missed something that already covers this, I did try look through other posts, but i didnt see anything that quite obviously covered this.

At present, it seems the only way to use Auto-GPT at present is GPT4 API Keys with OpenAI. Which is valid considering its early stages, but it shouldn't take much new code to add this functionality,

Required functionality can be summarized as: 1: Allowing users to choose another "Host/IP" other than OpenAI, TCP/IP Ports if required 2: The "Category" of the linked model (Language Model, Diffusion, TTS, Image to Text, STT, etc) 3: And specifying relevant API Key The first two being the only new features on this end.

Results would obviously vary, but the additional options would be good for the community.

A gentle colab with the folks working on Serge could go a long way for the self-hosting folks, Where as other parties might take a bit longer.

The Serge end of the problem requires them to solve their User profiles & authentication roadmap item, assuming that API integration would be part of the Authentication category.

The ideal end-state for such a feature would be, allowing auto-gpt to call multple models as part of a chain, depending on what is required.

Examples 🌈

Consider the prompt: "Write a short book of poems about AI, each poem should have a simple related graphic"

Action chain would resemble: Prompt -> GPT -> outline for book ->
GPT -> response -> Elaborate on each item of outline -> GPT -> firstItemResponse -> Stable Diffusion -> AssociatedGraphic Response -> CombinedResponse -> next item until complete -> book image -> project folder -> announce complete.

Creating a soundbite based on a prompt would have a similar flow, except instead of generating images, the Auto-GPT would feed the text prompt into a TTS ML via a similar chain of Hosts/API settings.

Motivation 🔦

This would enable those that are either Self Hosting GPT//LLaMAs, or leveraging other models through huggingface/similar parties, to make use of Auto-GPT, and empower their projects.

[vedangdate]

Could you elaborate on how it is different from 25. Except for chaining the models / understanding the intent of which model to use, I am not able to understand the difference.

[kagrith]

This goes beyond other LLMs, and "local", this is "Network" whether its another computer with more resources than where you are running autogpt from, or another docker, or a website analogous to OpenAI but not the same, whatever the linguistic complement of "Printing service" would be for Language models. VPS perhaps. Anyway.

The language of 25 appears to be focused on locally hosted models on the same machine as autogpt, opposed to reaching out toward open AI, my suggestion is to simply select dns hostname or IP/Port of the model, and the relevant API keys/formatting of the call. 25 is covered here under such a strategy by just entering "localhost:port" as the hostname/IP:Port

This would likely require making an /api/docs/ api call, interpreting the docs, and then building json templates... okay this might be more ambitious than i previously anticipated but serge serves an excellent example here. if "stub.domain.tld/api/docs/" were such a standard it would serve the community greatly, access should of course be behind an API key, but it would allow for a convenient framework for interconnecting models.

If you load up serge and go to localhost:8008/api/docs by default or whatever its been configured, to display docs written by the serge devs, and it is everything i've wanted from a SaaS/IaaS developer as far as docs before and seems simple enough that an LLM could parse it.

Attached image for reference.

Provided something like this were available, Being that it is for serge, this is why i thought serge should be a priority, as it already has support for other LLaMas, it can handle the passing of the prompt to the model

It should be easy enough to add the API for EasyDiffusion as well. Which adds another case for networked over local.

[github-actions[bot]]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions[bot]]

This issue was closed automatically because it has been stale for 10 days with no activity.