Multimodal feature for confluence (image summaries only)

[nisi99]

Description

Adds multimodal functionality to the Confluence connector: extracts all images that are in the attachment of a confluence page (and used on the page) and uses a (multimodal) LLM to create textual descriptions/summaries for each image. To improve the summary the XML of each confluence page is passed to the LLM as well as the title of the page and the name of the image. Each summary is stored as a new document, with the summary as the text of the section and the page of origin as link. Additional a label that marks the content of the document as a summary (label: is_image_summary) is created to enable users to directly recognise summaries as such.

With this approach, the first multimodal functions can be integrated into the Confluence connector without having to fundamentally adapt the retrieval or general structure of documents in Danswer.

Multimodality can be activated using the environment variable CONFLUENCE_IMAGE_SUMMARIZATION_MULTIMODAL_ANSWERING. If not used or false nothing will change, meaning no summaries will be generated and used for the answers. If CONFLUENCE_IMAGE_SUMMARIZATION_MULTIMODAL_ANSWERING is set false after indexing with summaries all summaries will be ignored in retrieval. Optionally users can set a custom system prompt (CONFLUENCE_IMAGE_SUMMARIZATION_SYSTEM_PROMPT) and user prompt (*

CONFLUENCE_IMAGE_SUMMARIZATION_USER_PROMPT*) as environment variables for the summarization. If not specified otherwise default prompts will be used.

How Has This Been Tested?

• (Unit) Tests: see scripts in backend/tests/multimodal_confluence
• Tested with own confluence space and AzureOpenAI (GPT-4o)

Accepted Risk (provide if relevant)

• If the summarization of an image fails due to content filters being triggered (or other errors) the indexing of the documents is restarted every x (default 30) minutes if not stopped manually. To avoid this problem we highly recommend setting CONTINUE_ON_CONNECTOR_FAILURE true, so the summaries for such images stay empty.

Related Issue(s) (provide if relevant)

N/A

Mental Checklist:

• All of the automated tests pass
• All PR comments are addressed and marked resolved
• If there are migrations, they have been rebased to latest main
• If there are new dependencies, they are added to the requirements
• If there are new environment variables, they are added to all of the deployment methods
• If there are new APIs that don't require auth, they are added to PUBLIC_ENDPOINT_SPECS
• Docker images build and basic functionalities work
• Author has done a final read through of the PR right before merge

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• [ ] This PR should be backported (make sure to check that the backport attempt succeeds)

[vercel[bot]]

@nisi99 is attempting to deploy a commit to the Danswer Team on Vercel.

A member of the Team first needs to authorize it.