Make Auto-GPT aware of it's running cost

Significant-Gravitas / AutoGPT

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

https://agpt.co

Other

168.72k stars 44.45k forks source link

Make Auto-GPT aware of it's running cost #6

Closed Torantulino closed 1 year ago

Torantulino commented 1 year ago

Auto-GPT is expensive to run due to GPT-4's API cost.

We could experiment with making it aware of this fact, by tracking tokens as they are used and converting to a dollar cost.

This could also be displayed to the user to help them be more aware of exactly how much they are spending.

0xcha05 commented 1 year ago

could use this? https://github.com/openai/tiktoken

Torantulino commented 1 year ago

My thoughts exactly.

If someone wants to tackle this.

Suggested Implementation:

[ ] ALL API calls should be centralised through a single method that counts tokens and adds to a dollar amount based on the model used.
[ ] Store this in its own file and remove openai imports from everywhere else so that the API can't be called accidentally.

Displaying

[ ] Total dollar amount spent should be constantly displayed to the user.
[ ] Total dollar amount spent could be displayed to the AI. Perhaps it could even be given a budget, and the dollar amount would count down as it spends money. This could create a sense of urgency in a HustleGPT style task.

If you'd like to work on this (please do!), assign it to yourself so we don't both work on the same thing.

0xcha05 commented 1 year ago

I can pick this, but first, I'll spend a little time on https://github.com/Torantulino/Auto-GPT/issues/4 .

0xcha05 commented 1 year ago

I'm going to start work on this now, @Torantulino . please assign this to me so others don't work on the same issue. I'll start with a fresh fork.

ryanpeach commented 1 year ago

I agree with all @Torantulino's ideas here. In that case, it might be a reason to use a real TUI instead of a simple set of print statements for this application. It could have a top-bar that contains running price for example.

I'd also make a flag in the cli that allows for rate limiting the application: Sleep 10 seconds between requests, for example. Some tasks the AI may be given may not require frequent polling. For example I'm using the bot for research, and notifications of major news events. Major news event searching may only need the bot to check the internet every day. This polling frequency option will be critical for giving some applications amazing speed, and others sort of dumb infrequent inexpensive speed.

masterismail commented 1 year ago

hey @0xcha05 are you currently working on this issue?

Vwing commented 1 year ago

I went ahead and implemented this in my own fork (branch "running_cost"). I'll submit a pull request later tonight.

Here it is in action, but also an example of it not working quite right.

running-cost-failure

I think I'll prompt engineer it a bit before submitting the pull request. I find that it works better if I send its remaining budget in the system message context, and yell at it if it goes over budget

    create_chat_message(
        "system", f"Your remaining budget is ${remaining_budget:.3f}" + (" BUDGET EXCEEDED! SHUT DOWN." if remaining_budget < 0  else "")),

Vwing commented 1 year ago

I'll submit a pull request later tonight.

I'm very close, now. Auto-GPT is beginning to behave with the appropriate behavior of hurrying up when the budget is nearly depleted, and shutting itself down when the budget is exceeded.

Too tired to finish this tonight, though. I'll give another update tomorrow night.

Vwing commented 1 year ago

Okay! It's working, now. It prints the running cost, the AI is aware of the running cost, and the user can optionally provide a budget (of which the AI is also aware, and behaves appropriately).

Please refer to my pull request. https://github.com/Torantulino/Auto-GPT/pull/762

Vwing commented 1 year ago

Woohoo, it's merged in! 🎉

@rob-luke would you be interested in tackling the final task of making the total running cost visible to the user?

See https://github.com/Significant-Gravitas/Auto-GPT/pull/762#issuecomment-1519147325

ntindle commented 1 year ago

@Vwing shoot me a DM on the discord :)

rob-luke commented 1 year ago

Thanks for tackling this issue @Vwing. I would be pleased to add the feature of making the running costs visible to the user, I can get that done this week. But if someone else starts to tackle it first, let me know so we don't duplicate work 🚀

Pwuts commented 1 year ago

Fixed in #762

rob-luke commented 1 year ago

@Pwuts The final aspect of the initial problem description is addressed in #3313 which isn't merged yet.

Boostrix commented 1 year ago

has anybody thought about exposing this in a shell prompt-style manner at runtime, per agent (task/job) ?