feat: Generate and add image captions to search index when image is ingested.

Azure-Samples / chat-with-your-data-solution-accelerator

A Solution Accelerator for the RAG pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences. This includes most common requirements and best practices.

MIT License

651 stars 324 forks source link

Closes #749

Purpose

This PR adds the following functionality when an image is uploaded to the knowledge base (when advanced image processing is enabled):

generates a caption for the image by calling the OpenAI service
generates the vector embeddings for the caption
includes the caption text and embeddings in the document uploaded to the search index

This will allow for improved search on images.

Does this introduce a breaking change?

[ ] Yes
[x] No

How to Test

Deploy Chat with your data with Advanced Image Processing enabled. Upload an image via the admin site. Then verify on the search index that the content and content_vector fields are populated with the appropriate caption and vector embeddings.

File	Stmts	Miss	Cover	Missing
code/backend/batch/utilities/helpers
env_helper.py	134	10	92%	222, 227–228, 231–233, 245, 249–251
llm_helper.py	42	11	73%	40–41, 50, 61–62, 73, 86–87, 94, 113, 121
code/backend/batch/utilities/helpers/embedders
push_embedder.py	74	0	100%
TOTAL	2434	684	71%

File

Stmts

Miss

Cover

Missing

code/backend/batch/utilities/helpers

env_helper.py

134

92%

222, 227–228, 231–233, 245, 249–251

llm_helper.py

73%

40–41, 50, 61–62, 73, 86–87, 94, 113, 121

code/backend/batch/utilities/helpers/embedders

push_embedder.py

100%

TOTAL

2434

684

71%

Tests	Skipped	Failures	Errors	Time
202	0 :zzz:	0 :x:	0 :fire:	11.817s :stopwatch:

Tests

Skipped

Failures

Errors

Time

202

0 :zzz:

0 :x:

0 :fire:

11.817s :stopwatch:

Azure-Samples / chat-with-your-data-solution-accelerator