Modular RAG - Query Analysis

[ThomasVitale]

Query Analysis

• Introduce Query Analysis Module
• Define QueryTransformer API and TranslationQueryTransformer implementation
• Define QueryExpander API and MultiQueryExpander implementation
• Support QueryTransformer in RetrievalAugmentationAdvisor (support for QueryExpander will be in the next PR together with the needed DocumentFuser API).

Improvements

• Refine Retrieval and Augmentation Modules for increased robustness
• Expand test coverage for both modules
• Define clone() method for ChatClient.Builder

Tests

• Introduce “spring-ai-integration-tests” for full-fledged integration tests
• Add integration tests for RAG modules
• Add integration tests for RAG advisor

Relates to #gh-1603

[markpollack]

Just some general thoughts, not necessary to resolve for this PR.

• The Query record. This is essentially the user text. What is this was instead the UserMessage so that it can take into account multi-modality (media) A QueryTransformer could for example retrieve images related to the text.
• The QueryAugmentor - what if we had as the signature QueryAugmentor extends BiFunction<Query, RetrievalContext, Query> where RetrievalContext is

  public class RetrievalContext {
    private final List<Document> documents;
    private final Map<String, Object> metadata;
  ...

this way additional information that can be used by an implementing class can be taken into account. The document is the core part, but there could be additional information to influence how the query gets augmented. Perhaps that would be passed in the constrcutor to the QueryAugmentor impl. Just looking to move away from types that don't have extensibility in the interface.

• Similar for DocumentRetriever, instead of List it returns a class RetrievedDocuments that contains the list plus additional data about those documents?

[markpollack]

I get test failures for tests org.springframework.ai.integration.tests due to the api-key not being set. Prob works in your environment but in my I don't have any spring.ai.openai.*properties set.

[markpollack]

merged in 263fe2fba7b6e940ac802e996db474d8909fe5d2