support gemini on vertexai

[arian81]

Feature request

As per documentation there's a package for Gemini support but this only works for Gemini API and doesn't work with Vertexai. https://python.langchain.com/docs/integrations/platforms/google However in the vertexai docs gemini is mentioned (for some reason gemini ultra ? ) even though when tried with geimini-pro (gemini-ultra is not out yet unless Langchain folks have connections at Google :) ) it's throwing an error indicating that model doesn't exist. https://python.langchain.com/docs/integrations/llms/google_vertex_ai_palm#multimodality Unknown model publishers/google/models/gemini-pro-vision; {'gs://google-cloud-aiplatform/schema/predict/instance/chat_generation_1.0.0.yaml': <class 'vertexai.language_models.ChatModel'>} (type=value_error)

Motivation

gemini has been out for a while and seemingly should be supported by langchain as they already made a whole package for it.

Your contribution

I would be willing to make a pr but I'm not even sure what's the issue since the docs supposedly mention that it should be already supported.

[dosubot[bot]]

🤖

It's great to hear that you're willing to contribute to LangChain by creating a PR! Your willingness to help improve the framework is much appreciated. It seems like there might be some confusion around the support for Gemini on VertexAI, but your initiative to investigate and potentially address the issue through a PR is commendable. If you encounter any specific challenges or need further clarification while working on the PR, feel free to reach out to the LangChain community for assistance. Your contribution will definitely help enhance the framework for the benefit of all users. Thank you for your willingness to contribute!

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[snayan06]

Facing a puzzling problem: some chains, like the "stuff chain," aren't responding in Gemini-pro, leading to worker timeouts. Even with verbose mode on, there's no extra info. It's like the thread enters the chain but doesn't return within 60 seconds, resulting in a Gunicorn worker timeout signal 9 kill. A bit perplexing!

this is how my chain is before entering i have .to_json() it ,

context_chain are:{'lc': 1, 'type': 'not_implemented', 'id': ['langchain', 'chains', 'combine_documents', 'stuff', 'StuffDocumentsChain'], 'repr': 'StuffDocumentsChain(verbose=True, llm_chain=LLMChain(verbose=True, prompt=PromptTemplate(input_variables=[\'context\', \'length\', \'question\'], template=\'\\nUse the following pieces of context to answer the question at the end.\\nReply with detailed answer in {length} and well formatted way.\\nMake use of bullet points wherever necessary. \\nDo not mention the word context in the answer.\\nIf the answer doesn\\\'t exist in the pieces of context, just reply only with "Sorry, I don\\\'t know the answer". \\nDon\\\'t try to make up an answer.\\nDo not mention the word context.\\n\\n{context}\\n\\nQuestion: {question}\\nHelpful Answer:\\n\'), llm=VertexAI(callbacks=[<faqsemanticsearcher.service.utils.stream_handler.LLMStreamHandler object at 0x7f42922d7460>], project=\'engati-1537172389163\', model_name=\'gemini-pro\', client=<vertexai.preview.generative_models._PreviewGenerativeModel object at 0x7f42922d7970>, client_preview=<vertexai.preview.generative_models._PreviewGenerativeModel object at 0x7f42922d7040>, credentials=<google.oauth2.service_account.Credentials object at 0x7f42922d7dc0>)), document_variable_name=\'context\')'}

and then after 60 second in journalctl if we check we get ....

Dec 28 11:25:04 pdr-appserver11.dev.engati.local bash[21691]: [2023-12-28 11:25:04 +0000] [21691] [CRITICAL] WORKER TIMEOUT (pid:21970)

and when we check the diff of timestamp btw logs it is exact 1 minute and 1 sec , also as mentioned above we dont get any verbose output as well despite in the chain we have mentioned it to be true....

[snayan06]

any update on vertexAI issues? or any threads where i can ask about this , as we currently have azure,openai integration with langchain and we would want to continue with that

[snayan06]

i was using the flask server with gevent , and that was causing the issue for me , i removed gevent and it started working very strange ...

[snayan06]

resolved this , with using this : https://github.com/grpc/grpc/pull/14561#issue-301487490 , thanks

[mchawa]

I think I'm having the same issue with gemini-pro with VertexAI with Flask also.

Getting the following error: NotFound('Publisher Model projects/xxxxxx/locations/us-central1/publishers/google/models/gemini-pro not found.')

@snayan06 I tried what you suggested but it didn't fix the issue for me.

A work around I found is to use the llm once it is created in a dummy way and it will work fine after that. e.g.: llm = VertexAI( model_name="gemini-pro", temperature=0.2, max_output_tokens=1024, verbose=True, ) llm.predict("Hi")

and then use the llm in other endpoints works fine.

I tried investigating the issue but couldn't find anything, hopefully someone checks this.

[snayan06]

can u share the error that u are getting u may be able to see it in the journalctl. and did u upgrade your version of grpc gevent ,grpcio i needed to upgrade that as well

[mchawa]

I'm using the following versions: gevent 23.9.1 grpcio 1.60.0

Also I'm using a Windows machine, so i'm not entirely sure where I can find the error log.

[Adi8885]

Hi @mchawa ,

I'm able to use Gemini-Pro with below setup : Python : 3.10 Langchain : 0.1.1 AIplatform : 1.38.1

pip install langchain==0.1.1 google-cloud-aiplatform==1.38.1

from langchain_google_vertexai import ChatVertexAI
llm = ChatVertexAI(model_name="gemini-pro")
llm.invoke("Who are you?")

>>> AIMessage(content='I am a large language model, trained by Google.')

Hope this helps