openvinotoolkit / model_api

Apache License 2.0

25 stars 16 forks source link

Async inference for cpp #177

Closed sovrasov closed 4 months ago

sovrasov commented 4 months ago

What does this PR do?

Asynchronous inference for cpp implementation is added:

Async queue
Batch mode emulation
Async tiling via batch mode
Example

Batch inference emulation with 100 inputs: