[FR]: Enable Opik to display additional media formats, including audio, PDF, and video.

[pleomax0730]

Proposal summary

Feature Request

Enable Opik to display additional media formats, including audio, PDF, and video.

Background

Opik currently supports only image display, which limits its flexibility for monitoring, testing, and evaluating multimodal LLM applications that may involve other data formats. Expanding support for audio, PDF, and video would allow users to fully leverage Opik’s capabilities across a broader range of use cases.

Proposed Use Cases:

• Audio Analysis: Track and evaluate audio-based LLM applications, such as voice transcription, voicebots, and sentiment analysis.
• PDF Document Evaluation: Facilitate assessment of document parsing models, PDF summarization, and question-answering systems on document data.
• Video Content Monitoring: Enhance capabilities for video-based LLM applications, such as video content analysis, summarization, and media captioning.

Benefits:

• Enhanced Multimodal Support: Broadens Opik’s applicability to multimodal applications, aligning with the needs of teams working across diverse LLM applications.
• Improved Traceability: Extends Opik’s tracing and feedback logging for all media types, ensuring comprehensive monitoring for complex, cross-modal projects.
• Unified Platform: Allows users to manage all media formats under a single platform, streamlining workflows.

Summary

Adding support for audio, PDF, and video display will make Opik a more versatile platform, suitable for a wide range of LLM applications beyond text and images. This enhancement will empower users to develop, evaluate, and monitor their applications seamlessly across all media types.

Motivation

Many existing solutions for LLM evaluation and monitoring are limited to text and image formats, with little to no support for other media types like audio, PDFs, or video. This lack of multimodal support forces teams to use multiple tools or rely on custom workarounds, creating friction in their workflows and hindering a comprehensive evaluation process.

By introducing audio, PDF, and video support, Opik could become the first open-source platform to offer complete multimodal monitoring and evaluation capabilities. This would make Opik highly attractive to teams working on complex applications that require seamless integration of various data formats, such as multimedia retrieval, interactive voice systems, and document processing pipelines.

Competitive Advantage:

Leading the way with these features would position Opik as a go-to solution for multimodal LLM applications, setting it apart from other evaluation and monitoring tools. This could significantly increase Opik’s user base by attracting organizations and researchers who need comprehensive, media-agnostic monitoring for their LLM projects.

[jverre]

Hi @pleomax0730 Thanks for the detailed request ! I really like this idea, we could introduce the concept of an attachment for a trace or span that allows you to log any additional data. Depending on the data type, we could introduce some ways to view the data in the UI.

Is there a specific type of data you would like to us to support first ? We have gotten requests for better PDF support so could be a good candidate to start with

[pleomax0730]

@jverre Thanks for your reply to my feature request! In terms of the current community, I think the best order might be PDF, then audio, and finally video.

[jverre]

Makes sense, I'll keep this ticket open. It's quite a big feature so we might not get to it straightaway