feat: Include vectorized text in search queries - Githubissues

Azure-Samples / chat-with-your-data-solution-accelerator

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

https://azure.microsoft.com/products/search

MIT License

656 stars 327 forks source link

feat: Include vectorized text in search queries #953

Closed cecheta closed 1 month ago

cecheta commented 1 month ago

Required by #750

Purpose

If advanced image processing is enabled, then the question will be vectorised with the same model as used for vectorising images
A second vector query is included when retrieving documents from AI search
Note that this is only for non-integrated vectorisation
Any retrieved images are not yet passed to the LLM, however the caption of the image is

Does this introduce a breaking change?

[ ] Yes
[x] No

How to Test

Enable advanced image processing
Deploy
Upload images
Ask questions based on images

github-actions[bot] commented 1 month ago

Coverage Report •

File	Stmts	Miss	Cover	Missing
code/backend/batch/utilities/helpers
azure_computer_vision_client.py	53	0	100%
azure_search_helper.py	57	0	100%
code/backend/batch/utilities/helpers/embedders
push_embedder.py	73	0	100%
code/backend/batch/utilities/search
azure_search_handler.py	62	2	96%	24, 30
integrated_vectorization_search_handler.py	68	3	95%	35, 135, 144
search.py	14	1	92%	16
search_handler_base.py	51	23	54%	16–18, 21–23, 27, 31, 35, 39, 43, 47, 51, 54–55, 57–59, 61–62, 65–66, 68
TOTAL	2460	672	72%

Tests	Skipped	Failures	Errors	Time
211	0 :zzz:	0 :x:	0 :fire:	11.437s :stopwatch:

cecheta commented 1 month ago

I jumped the gun 😬