neo4j-contrib / neo4j-apoc-procedures

Awesome Procedures On Cypher for Neo4j - codenamed "apoc"                     If you like it, please ★ above ⇧            
https://neo4j.com/labs/apoc
Apache License 2.0
1.68k stars 494 forks source link

vertexai gemini has a different URL, we should support that in the vertexai procedure #3939

Closed jexp closed 4 months ago

jexp commented 5 months ago

see:

curl -X POST \
    -H "Authorization: Bearer $(gcloud auth print-access-token)" \
    -H "Content-Type: application/json; charset=utf-8" \
    -d @gemini.json \ 
    "https://us-central1-aiplatform.googleapis.com/v1/projects/$PROJECT_ID/locations/us-central1/publishers/google/models/gemini-pro:streamGenerateContent"

currently we use :predict as suffix which doesn't seem to work anymore with gemini, also the request body changed a bit

we can probably have a new method called apoc.ml.vertexai.stream

that also works with the new body.

gemini-pro-vision supports also image payloads to e.g. describe or summarize images, not sure if we need that yet

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#request_body

POST https://{REGION}-aiplatform.googleapis.com/v1/projects/{PROJECT_ID}/locations/{REGION}/publishers/google/models/gemini-pro-vision:streamGenerateContent

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/gemini#gemini-pro-vision

We probably need that for gemini ultra anyhow.

jexp commented 5 months ago

Perhaps in addition we should add a generic URL method like we did in bedrock that allows to use more custom new model URLs

and perhaps an Image Vision procedure as well as we did for Bedrock.

https://cloud.google.com/vertex-ai/docs/generative-ai/model-reference/image-generation

it seems to support a GCS storage URL as optional parameter, so that the generated images are stored in GCS and an URL returned, we should support that.

And also see if something similar is available for the bedrock image-gen API with S3 and add it there too.