search

GoogleCloudPlatform / ai-on-gke

AI on GKE is a collection of examples, best-practices, and prebuilt solutions to help build, deploy, and scale AI Platforms on Google Kubernetes Engine
Apache License 2.0
225 stars 172 forks source link

Creating User for models that support gRPC requests. (Currently bespo… #629

Closed kfswain closed 5 months ago

kfswain commented 5 months ago

Currently bespoke for JetStream. As that is the only model using gRPC atm

liurupeng commented 5 months ago

if the grpc support is only targeting the JetStream, probably directly call it as JetStream-requester or Jet-client etc? btw, have you tried run benchmark with the change against the JetStream server?

kfswain commented 5 months ago

if the grpc support is only targeting the JetStream, probably directly call it as JetStream-requester or Jet-client etc? I'd like to keep as much code generic as possible to signal to any contributors to extend this code to support other gRPC Model Servers, instead of making a per server implementation. I'm hoping we can help drive towards a more generic proto since Google is the primary user of gRPC. I'm actually going to change JetStreamUser to instead be GrpcBenchmarkUser.

btw, have you tried run benchmark with the change against the JetStream server? Yes, I spun up my own cluster and JetStream implementation independently (did not use the provided stage-1/2 terraform). And then used the terraform in the locust directory to spin up the locust jobs. I can update the readme with the branching gRPC behavior