search

kestra-io / plugin-gcp

Apache License 2.0
9 stars 10 forks source link

VertexAI MultimodalCompletion task fails with an error #338

Closed shrutimantri closed 8 months ago

shrutimantri commented 8 months ago

Expected Behavior

VertexAI MultimodalCompletion task should run successfully.

Actual Behaviour

VertexAI Multimodal task fails with the following error:

2024-03-17 22:55:12.552 • Using service account: smantri-kestra-20230317@empirical-realm-374307.iam.gserviceaccount.com
2024-03-17 22:55:13.083io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/html; charset=UTF-8
headers: Metadata(:status=404,content-type=text/html; charset=UTF-8,referrer-policy=no-referrer,date=Sun, 17 Mar 2024 17:25:13 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,content-length=1627)
DATA-----------------------------
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/google.cloud.aiplatform.v1.PredictionService/StreamGenerateContent</code> was not found on this server.  <ins>That’s all we know.</ins>
2024-03-17 22:55:13.083UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/html; charset=UTF-8
headers: Metadata(:status=404,content-type=text/html; charset=UTF-8,referrer-policy=no-referrer,date=Sun, 17 Mar 2024 17:25:13 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,content-length=1627)
DATA-----------------------------
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/google.cloud.aiplatform.v1.PredictionService/StreamGenerateContent</code> was not found on this server.  <ins>That’s all we know.</ins>
2024-03-17 22:55:13.083com.google.api.gax.rpc.UnimplementedException: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/html; charset=UTF-8
headers: Metadata(:status=404,content-type=text/html; charset=UTF-8,referrer-policy=no-referrer,date=Sun, 17 Mar 2024 17:25:13 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,content-length=1627)
DATA-----------------------------
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/google.cloud.aiplatform.v1.PredictionService/StreamGenerateContent</code> was not found on this server.  <ins>That’s all we know.</ins>

    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:108)
    at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:41)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:86)
    at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
    at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
    at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:84)
    at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:148)
    at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
    at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
    at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
    at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:570)
    at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
    at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
    at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.base/java.lang.Thread.run(Unknown Source)
    Suppressed: java.lang.RuntimeException: Asynchronous task failed
        at com.google.api.gax.rpc.ServerStreamIterator.hasNext(ServerStreamIterator.java:105)
        at com.google.cloud.vertexai.generativeai.ResponseStreamIteratorWithHistory.hasNext(ResponseStreamIteratorWithHistory.java:37)
        at com.google.cloud.vertexai.generativeai.ResponseHandler.aggregateStreamIntoResponse(ResponseHandler.java:104)
        at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:547)
        at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:612)
        at com.google.cloud.vertexai.generativeai.GenerativeModel.generateContent(GenerativeModel.java:560)
        at io.kestra.plugin.gcp.vertexai.MultimodalCompletion.run(MultimodalCompletion.java:95)
        at io.kestra.plugin.gcp.vertexai.MultimodalCompletion.run(MultimodalCompletion.java:33)
        at io.kestra.core.runners.Worker$WorkerThread.run(Worker.java:710)
Caused by: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: text/html; charset=UTF-8
headers: Metadata(:status=404,content-type=text/html; charset=UTF-8,referrer-policy=no-referrer,date=Sun, 17 Mar 2024 17:25:13 GMT,alt-svc=h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,content-length=1627)
DATA-----------------------------
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 404 (Not Found)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/></a>
  <p><b>404.</b> <ins>That’s an error.</ins>
  <p>The requested URL <code>/google.cloud.aiplatform.v1.PredictionService/StreamGenerateContent</code> was not found on this server.  <ins>That’s all we know.</ins>

    at io.grpc.Status.asRuntimeException(Status.java:533)
    ... 17 more

I am not sure if I am missing something here.

Steps To Reproduce

  1. Execute the flow in the sample example with appropriate values.
  2. The execution fails with an error.

Environment Information

Example flow

id: vertexai-multimodal-completion
namespace: dev
tasks:
  - id: "multimodal_completion"
    type: "io.kestra.plugin.gcp.vertexai.MultimodalCompletion"
    region: "eu-west3"
    projectId: "<project-id>"
    serviceAccount: "{{ secret('GCP_SERVICE_ACCOUNT_JSON') }}"
    contents:
      - content: Please tell me a joke
loicmathieu commented 8 months ago

Hi, This is because the service for multi-modal completion is not available in eu-west3, if you do the same in the region us-central1 it works