Support for VertexAI for Gemini LLM and Embedding

[tsibg]

I am interested in support for Gemini models, preferably on VertexAI by Google Cloud Platform. The Gemini 1.5 Pro supports up to 1M Context, which could open a bunch of possibilities for grounding and RAG. Further more it is multimodal (text, images, audio, PDFs, code, videos).

Resources for the integration:

LLM and Embedding models on VertexAI:

• https://cloud.google.com/vertex-ai/generative-ai/docs/learn/models

Langchain LLM support for VertexAI and all Gemini models:

• https://js.langchain.com/docs/integrations/llms/google_vertex_ai

Langchain Embeddings on VertexAI (Expirimental): Supports both textembedding-gecko and multimodalembedding@001

• https://js.langchain.com/docs/modules/data_connection/experimental/multimodal_embeddings/google_vertex_ai

I'm currently working on the integration, it looks pretty straight forward. Will also submit a PR, if the authors are interested.

P.S. Thanks for creating this library, well structured and easy to use.

[adhityan]

Thank you @tsibg both for engaging and also for the PR. Let's get it merged in.

[tsibg]

Thanks for merging it.

Just a few ideas for future implementations:

• Gemini also supports multimodal. It can directly accept PDF, video, images. Maybe it is bit out of scope for this library, but in case anyone needs it, the integration can be updated.
• Same for the Embeddings: the LangChain's module have different class for multi modal embeddings, it could be implemented. However, not sure how it will work with the rest of the library.

Closing this issue as my use case is fully covered with the merge. Thanks again for the fast collab!

[adhityan]

Published new version 0.0.72 with merged changes.

I like the idea of natively supporting multimodal. RAG is still useful in a multimodal setup. I will start a discussion thread on the best way to go ahead with support for this in the library.