search

Portkey-AI / gateway

A Blazing Fast AI Gateway with integrated Guardrails. Route to 200+ LLMs, 50+ AI Guardrails with 1 fast & friendly API.
https://portkey.ai/features/ai-gateway
MIT License
5.87k stars 403 forks source link

[Provider] Add support for Google Vertex AI #10

Closed vrushankportkey closed 5 months ago

vrushankportkey commented 11 months ago

Docs: https://cloud.google.com/vertex-ai/docs/reference/rest

flexchar commented 6 months ago

I'd like this too. And I'm afraid I will find myself tinkering on this one in the near future. Implementing here would do so much more good for the community and for the projects I'd like to integrate.

The thing is that we would likely need https://github.com/googleapis/nodejs-vertexai.

One issue I can already foresee is that Goggle's Service Account authentication is not supported on Cloudflare Workers (it's where I currently run my very deployment of the gateway and some other software. I tried to get it working but I end up hitting all the corners. Could be a skill issue. However this would work fine if deployed on classic runtime.

We could also use the Rest API using access token but Google's access tokens expire within 60 minutes. So it's unlikely that whose who can generate it will need a gateway (thou not impossible). And those who cannot generate will likely reach a point of frustration.

If anyone who has more experience with Google's SA and edge runtimes could shine, would be great to hear thoughts. I recently discovered that OCR features of Gemini 1.0 Pro Vision are superior to GPT-4V, so let alone what will Gemini 1.5 bring (waiting for access).

roh26it commented 6 months ago

Thanks for the note @flexchar !

Gemini has been relatively easy to work with and that is supported within the Gateway. So, if you're looking to use Gemini 1.0 Pro Vision or even other Gemini models, they work well.

Vertex AI has auth and geo challenges that we're still trying to figure out. Would love if you could take a look or suggest the way forward.

flexchar commented 6 months ago

The problem for me, perhaps the most essential point, that I failed to mention, is that the Gemini API (aka, Google AI Studio API) is blocked in EU. So I can only access if I use a VPN to, for example, a city in USA.

Vertex AI, on the other hand, does not impose geo restrictions based on IP address. This is rather mysterious for me as to why...

@roh26it