search

fjrdomingues / autopilot

Code Autopilot, a tool that uses GPT to read a codebase, create context and solve tasks.
584 stars 63 forks source link

gpt-3.5-turbo behaving differently #119

Open fjrdomingues opened 1 year ago

fjrdomingues commented 1 year ago

Openai must have changed something on the model and now the model "refuses" to answer in JSON. Also tested on chatgpt, with the same result. Before it was working almost 100% of the time.

We may be able to solve this by using the "gpt-3.5-turbo-0301" version of the model. Haven't tested it yet.

fjrdomingues commented 1 year ago

This doesn't seem to be a problem using the new implementation with langchain output format, which is great 👍 Still a warning that model can change, causing unpredictable results. This is why openai launches versions of the models like gpt-3.5-turbo-0301 https://platform.openai.com/docs/models/gpt-3-5

hatkyinc2 commented 1 year ago

I've just run a couple of tests on my end I have no issues generating JSON in either chatGPT or playground gpt-3.5-turbo nor the gpt-3.5-turbo-0301

I can see how this could be a problem, generally, any 3rd party agent can change anything we count on at any time, which is one reason I like the local models direction, it's a frozen state.

🤷 Sounds like gpt-3.5-turbo-0301 will be short-lived, but it might be a reminder to generalize the model support more, so it's easy to test. Currently, the biggest hardcoded part is the pricing I think.

hatkyinc2 commented 1 year ago

Please add a unit test to cover this issue if it's still relevant @fjrdomingues or should we just close this for now?