Setting the max_tokens to 4096 will output always an error like this:
I said for you to remove every mention on the code about max_tokens or tokens parameters so that it defaults to Inf (4096 - prompt tokens). Inf is not infinite, Inf equals to 4096 - prompt tokens, so that no one needs to keep changing the ata.toml to reduce or to increase the max_tokens limit. It will automatically make the calculation:
Allowing OpenAI to choose the maximum response length is like allowing Shell to decide how much fuel I'm going to tank. I've set it to 2048 now as a reasonable default.
Hi
I think you did a mistake in your last commit.
Setting the max_tokens to 4096 will output always an error like this:![Screenshot_2023-03-08_14-47-16](https://user-images.githubusercontent.com/9296780/223790694-618cdcca-4a23-4949-ba90-bdcf3c03974d.png)
I said for you to remove every mention on the code about max_tokens or tokens parameters so that it defaults to Inf (4096 - prompt tokens). Inf is not infinite, Inf equals to 4096 - prompt tokens, so that no one needs to keep changing the ata.toml to reduce or to increase the max_tokens limit. It will automatically make the calculation:
Who explained that to me was the awesome dev Jack Wong on a issue in his incredible app: https://github.com/jw-12138/davinci-web/issues/8