search

triton-inference-server / fil_backend

FIL backend for the Triton Inference Server
Apache License 2.0
72 stars 36 forks source link

Add batch stats reporting to Triton #397

Closed hcho3 closed 4 months ago

hcho3 commented 4 months ago

Use TRITONBACKEND_ModelInstanceReportBatchStatistics to report statistics about batching.

Example: https://github.com/triton-inference-server/tensorflow_backend/blob/515466c5760612aad0458d5e4f832c4114163271/src/tensorflow.cc#L2548-L2553

hcho3 commented 4 months ago

Already implemented in Rapids-Triton.