Any example of triton-vllm in c++?

triton-inference-server / client

Triton Python, C++ and Java client libraries, and GRPC-generated client examples for go, java and scala.

BSD 3-Clause "New" or "Revised" License

521 stars 225 forks source link

Any example of triton-vllm in c++? #678

Closed tricky61 closed 1 month ago

tricky61 commented 1 month ago

I use the tc::InferenceServerGrpcClient.Infer but got the error of not supported support models with decoupled transaction policy. when I use tc::InferenceServerGrpcClient.AsyncInfer, no error message but no request in the server. but the server log has error: Infer failed: ModelInfer RPC doesn't support models with decoupled transaction policy which API should I use?