[ENHANCEMENT] Support for Multi-Modal RAG Tracing and Evaluation

[hakantekgul]

It would be great if Phoenix supported Multi-Modal applications, such as a Multi-Modal RAG. There are a couple of features to consider in order to fully support this:

MM RAG Tracing:

• It would be very helpful if images were rendered on Phoenix UI, whether the image exists in a response, context, or response. Given image paths, rendering the images would provide a lot of visibility.
• I think this is already supported but testing the whole framework against LMMs such as GPT-4V would be helpful. Example screenshot from Phoenix:

MM RAG Evaluation:

• Having Multi-Modal evaluators using LMMs as a Judge could be very helpful. Given image_paths, reference text etc, having MultiModalRelevancy and MultiModalHallucination evals etc. under Phoenix evals could bring a lot of value.
• Right now, LlamaIndex provides MM Evals:

[axiomofjoy]

Thanks @hakantekgul

[mikeldking]

Hey @hakantekgul appreciate the feedback. We probably have a bit too much on our plate right now to tackle at the current moment but keep us honest on the priority of this.

[stefanadelbert]

Adding support for working with multimodal models would be a huge step forward, e.g. being able to visualise images in prompt inputs (base64 encoded PNG or JPEG) in the Phoenix UI Trace Details. rather than seeing the base64 encoded text as shown in this screenshot.

[mikeldking]

Adding support for working with multimodal models would be a huge step forward, e.g. being able to visualise images in prompt inputs (base64 encoded PNG or JPEG) in the Phoenix UI Trace Details. rather than seeing the base64 encoded text as shown in this screenshot.

Soon:)