This introduces the "isolated batcher", a batcher that keeps batches isolated by one another by not merging.
The idea is that this can be paired with the dynamic scheduler, so they dynamic scheduler can be used without merging multiple customer's data into the same batch.
The docker file builds the batcher and installs it to:
/opt/tritonserver/batching/isolated_batcher/libtriton_isolated_batcher.so
I have yet not tested enabling this batcher as it needs a separate model config change.
This introduces the "isolated batcher", a batcher that keeps batches isolated by one another by not merging.
The idea is that this can be paired with the dynamic scheduler, so they dynamic scheduler can be used without merging multiple customer's data into the same batch.
The docker file builds the batcher and installs it to:
/opt/tritonserver/batching/isolated_batcher/libtriton_isolated_batcher.soI have yet not tested enabling this batcher as it needs a separate model config change.