Duplicated URL Results in `llamaindex-db`

[k-allagbe]

Description

When using AI Lab LlamaIndex Search, it is possible to receive multiple search results that reference the same URL. This behavior contrasts with Azure AI Search, where each result corresponds to a unique URL.

Tasks

• [x] Investigate the cause of duplicate URL results in AI Lab LlamaIndex Search responses: here
• [x] Explore configuration options or filters in AI Lab LlamaIndex Search to prevent duplicate URLs.
• [x] Implement a solution to ensure unique URLs in search results, mirroring the behavior seen with Azure AI Search.
• [x] Add unit tests to verify that search results contain unique URLs.
• [x] Document the changes and the configuration settings used to achieve the desired result.

Acceptance Criteria

• No search result set from AI Lab LlamaIndex Search should contain duplicate URLs.
• The implemented solution should not significantly impact the performance or response time of the search.
• All unit tests related to search results must pass, confirming the uniqueness of URLs in the results.

[k-allagbe]

Try grouping by URL.

[k-allagbe]

Reopening.

"What are acceptable methods for disposing of food?"