search

simonw / llm-gemini

LLM plugin to access Google's Gemini family of models
Apache License 2.0
86 stars 6 forks source link

Support for Gemini via Vertex API #8

Open justyns opened 2 months ago

justyns commented 2 months ago

Thanks for making llm and all of your work around it!

If I understand correctly, this plugin, llm-gemini, only works with the gemini api through Google AI Studio. But it doesn't work with the gemini models through Google Vertex AI which uses different endpoints.

It's kind of confusing because google has two python libraries that are really similar. google-cloud-aiplatform is the one that works with Vertex. Code example: https://cloud.google.com/vertex-ai/generative-ai/docs/start/quickstarts/quickstart-multimodal

Do you think llm-gemini could support using the Vertex API in addition to the aistudio api? (I'm not sure what to call the non-vertex api)

My use case is pretty simple: At work, I can access the Vertex AI API through GCP, but not AI Studio. Vertex also doesn't use your data for training purposes by default.

simonw commented 2 months ago

I'll be honest, I hadn't figured out the difference between AI Studio and Vertex API myself! Thanks for helping clarify.

I can see a few options:

  1. Ship a separate plugin, llm-gemini-vertex (or llm-vertex even, no need for that to be limited to just Gemini models) which uses the Google Vertex AI API
  2. Teach llm-gemini to call both. I'm not sure how best to have it decide which APi to use though - a couple of options:
    1. You set a different secret for each one - if you set gemini (the current) or aistudio (a new one) then it uses AI Studio, if you set vertex then it uses the Vertex API
    2. Register multiple models names - gemini-pro and gemini-1.5-pro-latest and gemini-pro-vertex and gemini-1.5-pro-latest-vertex

I think the llm-vertex plugin may be the best of these options.

justyns commented 2 months ago

I'll be honest, I hadn't figured out the difference between AI Studio and Vertex API myself! Thanks for helping clarify.

No promises that I'm 100% correct, the docs aren't great at clarifying so I could've missed a way to get things working :)

After thinking about it, I agree with you that llm-vertex probably makes the most sense. I've only used the Gemini models in vertex, but I know you can train your own models or just use models other than Gemini, though I don't think they're all "chat" models.

For auth, using google's sdk is probably easiest. Usually an app would use either auth set up by the gcloud command, or a .json file pointing to service account in GCP. I couldn't find a way to use just an api key or token.

justyns commented 2 months ago

Hey @simonw , I took a crack at this tonight and created a plugin based on https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html:

It works so far, but I left a few TODOs I wasn't sure about. Like should I register all models on an account? And should I add a prefix to the model name like vertex/gemini-1.5-pro-preview-0409 instead of just gemini-1.5-pro-preview-0409 to avoid possibly clashing with the llm-gemini plugin?

edit: And also, I'm using environment variables to set the gcp project id/location. Would it be better to use llm keys set vertex_project_id or something similar instead?