fix: Add `max_tokens` to chat completions

Azure-Samples / chat-with-your-data-solution-accelerator

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

https://azure.microsoft.com/products/search

MIT License

651 stars 324 forks source link

fix: Add `max_tokens` to chat completions #934

Closed cecheta closed 1 month ago

cecheta commented 1 month ago

Purpose

GPT-4 has a default max tokens of 16, meaning the captions generated were very small. This PR adds the AZURE_OPENAI_MAX_TOKENS to the chat completions call, so it will default to 1000 tokens, and can be conifgured.

Does this introduce a breaking change?

[ ] Yes
[x] No

How to Test

Enable Advanced Image Processing
Upload image
View caption in Azure Search

adamdougal commented 1 month ago

Do we know why that's the default? Is it more expensive?

github-actions[bot] commented 1 month ago

Coverage Report •

File	Stmts	Miss	Cover	Missing
code/backend/batch/utilities/helpers
llm_helper.py	42	11	73%	40–41, 50, 61–62, 73, 86–87, 94, 113, 121
TOTAL	2434	684	71%

Tests	Skipped	Failures	Errors	Time
202	0 :zzz:	0 :x:	0 :fire:	19.011s :stopwatch:

cecheta commented 1 month ago

Do we know why that's the default? Is it more expensive?

Not sure, seems to be just for vision