dynamic_batching with model config

triton-inference-server / fastertransformer_backend

BSD 3-Clause "New" or "Revised" License

411 stars 134 forks source link

Closed hajime9652 closed 2 years ago

hajime9652 commented 2 years ago

It's very convenient and helpful. Thank you.

Is it possible to use dynamic_batching on 1 node 1 gpu?

byshiue commented 2 years ago

Yes. That's feature of triton backend.

byshiue commented 2 years ago

Close this bug because it is inactivated. Feel free to re-open this issue if you still have any problem.