ModelPipelined allows for parallel workers processing and post-processing a model. This way threads can be spawned to take advantage of spare compute and memory on whichever part of processing bottlenecks, increasing the throughput at minimal latency cost. Inactive model forwarding threads will always claim the most recent non-redundant data to process.
Other commits are generally in service of this struct. Benchmarks for pipelined execution and each part of execution were added, to prove that inter-thread communication overhead is minor and make on-Jetson tuning measurements feasible.
ModelPipelinedallows for parallel workers processing and post-processing a model. This way threads can be spawned to take advantage of spare compute and memory on whichever part of processing bottlenecks, increasing the throughput at minimal latency cost. Inactive model forwarding threads will always claim the most recent non-redundant data to process.Other commits are generally in service of this struct. Benchmarks for pipelined execution and each part of execution were added, to prove that inter-thread communication overhead is minor and make on-Jetson tuning measurements feasible.