Vector search with HTML text and images together

[MatteoAntonini]

I've seen several examples in the repository with separate text and images, but if I wanted to perform a RAG on HTMLs containing both text and images, what would be the best approach? Currently, I've only tried with the text present in the HTML, creating a vector with a content field containing the HTML text. In particular, I'm not sure whether to include images beforehand and create a single skillset, or keep them separate and then merge the two indexes for text and images.

[farzad528]

Hey @MatteoAntonini thanks for flagging this question! When working with HTML content that includes both text and images, you need to handle text and images differently because they are inherently different types of data. Here are some things you could do:

• Extract text and image features separately: You can create two separate skillsets, one for text extraction and another for image analysis. The text extraction skillset can use the built-in text extraction capabilities of Azure AI Search to extract text from the HTML, while the image analysis skillset can use computer vision APIs to extract features from the images. Once you have extracted the features, you can merge them into a single index.
• Use a custom skill to extract text and image features: You can create a custom skill that combines text extraction and image analysis into a single step. The custom skill can use libraries such as Beautiful Soup or Selenium to extract text from the HTML and OpenCV or Pillow to extract features from the images. The output of the custom skill can be a single vector that combines text and image features.
• Use a pre-trained model for HTML analysis: You can use a pre-trained model such as HTML-BERT that is specifically designed for HTML analysis. These models can extract both text and image features from the HTML and generate a single vector for RAG analysis. Note, I haven't personally used this approach before but it could work!

I think regardless of the approach you choose, it's important to consider the relevance of the images in the HTML for your specific use case. If the images are highly relevant to the search query, it may be beneficial to include them in the RAG analysis. However, if the images are not relevant, they may add noise to the analysis and reduce the overall relevance of the search results.

Hope this helps!