search

Pythagora-io / gpt-pilot

The first real AI developer
Other
29.56k stars 2.94k forks source link

Ability to use GPT 3.5 Turbo #31

Closed Zate closed 1 year ago

Zate commented 1 year ago

I keep hitting the limits of my GPT4 API access, and I'd love to have the capacity to change over to using GPT3.5

Has any testing been done with that? is it not delivering what is needed or would it be possible to allow the swapping of modesl via a config item perhaps? Some things I might need the power and complexity of GPT4, but I suspect we could likely make do with GPT 3.5 in some situations perhaps?

sander110419 commented 1 year ago

I've made a PR to enable the selection of the model. We aware that GPT3.5 provides way worse code, you're better off using the Azure endpoint and using the gpt4-32k model.

CyKiller commented 1 year ago

Not sure how to fully map out handling prioritized model switching for gpt-pilot, but introducing a rate-limiting mechanism in llm_connection.py to manage API requests to both GPT-4 and GPT-3.5 models can be done. We can try implementing counters for Tokens Per Minute (TPM) and Requests Per Minute (RPM) / then add logic to switch between the two models. These counters would enable dynamic model switching, allowing the system to choose the most appropriate model based on current usage and rate limits. Additionally, we can incorporate an exponential mechanism strategy to handle "429: Too Many Requests" errors. This involves a while-loop that retries the API request, doubling the wait time with each subsequent retry, up to a maximum number of retries. The contributor's input is better on how to target this, it's good for emulating instant gratification but in no form is it a true model switching formula that could be a nice fit for gpt-pilot.

coolxeo commented 1 year ago

Just created a PR that could solve it in a simple way https://github.com/Pythagora-io/gpt-pilot/pull/39

I recommend to use gpt-3.5-turbo-16k-0613 model as has high capacity

coolxeo commented 1 year ago

If you are having problems with rate limited you could try to downgrade to GPT-3 that doesn't have so many restrictions

To make the change, just setup your environment variable in your .env file to whatever model you want to use OPENAI_MODEL=gpt-3.5-turbo-16k-0613

Reference of GPT-3 models https://platform.openai.com/docs/models/gpt-3-5

Thanks @zvone187 for merging my PR 🤗

zvone187 commented 1 year ago

Happy to do so - thank you for the PR 🙏