Stablize compilation time measurement in benchmarks

iree-org / iree

A retargetable MLIR-based machine learning compiler and runtime toolkit.

http://iree.dev/

Apache License 2.0

2.56k stars 568 forks source link

Stablize compilation time measurement in benchmarks #11922

Closed antiagainst closed 3 weeks ago

antiagainst commented 1 year ago

Right now we measure the compilation time for compiling various models. Though the numbers reported can vary a lot. For example, this one ranges from 60s to 140s; similarly for CPU. With such a level of noisiness we cannot draw any meaningful information out of it and it's actually a distraction. We'd need to figure out ways to make the numbers more stable.

pzread commented 4 months ago

Unassigned myself as I'm not working on this currently