Adds ability to record run times in Pipeline and Ops, given inference state
Problem
Currently, we dont have a way to get the run time info in the pipeline and in the operator
Solution
We add timer manager as an attribute to the Pipeline, and add Timer logic into the InferenceState as an attribute (see tests)
Design
We re use the timer attribute in the inference state. All run-times will be saved there and will be written to the timer manager once the Pipeline run ends.
Usage
Given access to inference_state
with inference_state.time(id="foo"):
time.sleep(1)
Testing
Checks that the runtime from ops and Pipeline is saved into the timer manager
multi-threaded check
The runtime without saving the measurements and the runtime saving the measurements are checked to be same
The expected time that is saved is correct
no timer, no wait : 2.95375657081604
no timer, 0.2 wait: 146.53509402275085
pipeline timer, no wait : 3.020995855331421, 2.8153765201568604
pipeline timer, 0.2 wait: 150.66221618652344
pipeline + ops timer , no wait : 2.9319729804992676 3.0496866703033447 3.1929404735565186
pipeline + ops timer, 0.2 wait: 154.01117300987244
from deepsparse import Pipeline
model_path = "hf:neuralmagic/TinyLlama-1.1B-Chat-v0.4-pruned50-quant-ds"
Description
Adds ability to record run times in Pipeline and Ops, given inference state
Problem
Currently, we dont have a way to get the run time info in the pipeline and in the operator
Solution
We add timer manager as an attribute to the Pipeline, and add Timer logic into the InferenceState as an attribute (see tests)
Design
We re use the timer attribute in the inference state. All run-times will be saved there and will be written to the timer manager once the Pipeline run ends.
Usage
Given access to inference_state
Testing
pipeline = Pipeline.create( task="text_generation", model_path=model_path, engine_type="deepsparse", # onnxruntime internal_kv_cache=True, # False # ) import time start =time.time() output = pipeline(
prompt= ["The sun shined"],
) rt = time.time() - start print(rt) breakpoint()