Open whzghb opened 2 months ago
@Tabrizian can you help answer here.
@whzghb Yes, when deploying stateful models you need to make sure to send all the request with the same correlation ID to the same Triton instance.
@whzghb Yes, when deploying stateful models you need to make sure to send all the request with the same correlation ID to the same Triton instance.
@Tabrizian That means it depends on the client, not the server, right?
When tritonserver is deployed as multiple pods, for stateful models, does traffic have to be sent to the pod of the last inference request? If so, how is it done?