Feature Request: Add `VoyageAI Embedding Models` support to Haystack (v2.0x)

[awinml]

Add VoyageAI Embedding Models support to Haystack (v2.0x)

• VoyageAI launched new embedding models that claim state-of-the-art performance on the MTEB Benchmark, outperforming popular embedding models like BAAI - bge and OpenAI - text-embedding-ada-002.
• Released two models - voyage-01 and voyage-lite-01. The performance comparison shown by them using voyage-01can be seen below.

Embedding models released:

Currently supports two models:

• voyage-01 Context Length (tokens): 4096 Embedding Dimension: 1024
• voyage-lite-01
  Context Length (tokens): 4096 Embedding Dimension: 1024

VoyageAI also plans to offer embedding models tailored for coding and finance, with more domains on the horizon:

• voyage-xl-01
• voyage-code-01
• voyage-finance-01

More information about their embedding models can be found on their Embeddings documentation.

Simple usage example using the Python SDK:

A complete working example to perform Semantic Search uses these embedding models can be found in this Colab Notebook.

Python SDK API Documentation: https://docs.voyageai.com/embeddings/#via-voyage-python-library

import voyageai 
from voyageai import get_embedding

voyageai.api_key = "[ Your Voyage API KEY ]"  # add your Voyage API KEY

documents = [
    "The Mediterranean diet emphasizes fish, olive oil, and vegetables, believed to reduce chronic diseases.",
    "Photosynthesis in plants converts light energy into glucose and produces essential oxygen.",
    "20th-century innovations, from radios to smartphones, centered on electronic advancements.",
    "Rivers provide water, irrigation, and habitat for aquatic species, vital for ecosystems.",
    "Apple’s conference call to discuss fourth fiscal quarter results and business updates is scheduled for Thursday, November 2, 2023 at 2:00 p.m. PT / 5:00 p.m. ET.",
    "Shakespeare's works, like 'Hamlet' and 'A Midsummer Night's Dream,' endure in literature."
]

# Embed the documents
embeddings = get_embeddings(documents, model="voyage-01")

Describe the solution you'd like

Integrate VoyageAI Embeddings with Haystack (v2.0x), specifically a VoyageTextEmbedder and VoyageDocumentEmbedder using the VoyageAI python SDK.

[anakin87]

Hello, @awinml! And sorry for the late reply...

It would be a nice integration to have.

If you would like to contribute to this integration, I would propose to:

• create your repository for this integration (similar to other integrations you have created in the past)
• we and you can promote it through the haystack-integrations page.

Feel free to let us know what you think. Thanks!

[awinml]

That sounds good! These models are new and the API access is not fully open yet, so it might take a while for them to become popular.

I will create my own repository for this integration in the meantime and create a PR in haystack-integrations once its ready.

Thanks! @anakin87

[awinml]

@anakin87 I have created a repository (voyage-embedders-haystack) for the integration and also opened a PR (https://github.com/deepset-ai/haystack-integrations/pull/85) in haystack-integrations. Thanks for your help!

[anakin87]

Great! We will review your PR in haystack-integrations...