search

neuralmagic / deepsparse

Sparsity-aware deep learning inference runtime for CPUs
https://neuralmagic.com/deepsparse/
Other
2.97k stars 171 forks source link

V2 Timer Manager #1472

Closed horheynm closed 7 months ago

horheynm commented 8 months ago

Blocked on https://github.com/neuralmagic/deepsparse/pull/1471

Description

Add Timer to middleware, which will record the runtime of call in Pipeline and Operator

Problem

No existing way to get the run time of the callables

Solution

Add TimerMiddleware to the existing middleware

Design

General middleware implementation. Record start time, run the callable, get the end time and the overall run-time

Usage

Check tests (/tests/deepsparse/middlewares/test_timer_middleware_pipeline.py)


 middlewares = [
        MiddlewareSpec(TimerMiddleware),  # for timer
    ]

    ops = [AddOneOperator(), AddTwoOperator()]

    AddThreePipeline = Pipeline(
        ops=ops,
        router=LinearRouter(end_route=2),
        schedulers=[OperatorScheduler()],
        continuous_batching_scheduler=ContinuousBatchingScheduler,
        middleware_manager=MiddlewareManager(middlewares),
    )

    pipeline_input = IntSchema(value=5)
    pipeline_output = AddThreePipeline(pipeline_input) # output is 8

    pipeline_measurements: List[
        defaultdict
    ] = AddThreePipeline.timer_manager.measurements
    measurements = pipeline_measurements[0]

>> measurements
defaultdict(<class 'list'>, {'AddOneOperator': [9.202957153320312e-05], 'AddTwoOperator': [3.9577484130859375e-05], 'total': [0.0009074211120605469]})

Testing

Checks if timings are measured, and exists as expected

mgoin commented 7 months ago

could you please make the TimerManager in time.py have a __repr__() defined like the one in timer.py? this will let it have a useful string representation since currently it just prints the raw object:

(Pdb) model.timer_manager
<deepsparse.utils.time.TimerManager object at 0x7fd9c6987be0>