Open flexwang opened 1 year ago
In the ensemble model example for gpt, can I change the fastertransformer model to a decoupled model and enable streaming on the client side?
fastertransformer
decoupled
+1
Answer is yes
Looks like only FT backend support stream, however python backend does not.
In the ensemble model example for gpt, can I change the
fastertransformer
model to adecoupled
model and enable streaming on the client side?