xyhuang
commented
6 years ago Thanks for bringing this up! I think we may start with building a traffic generator/simulator and result collector, and connect with tfserving/seldon as a POC. However to make it really useful it would be good to see what problem are we looking to solve through serving benchmarks. Some questions i have:
- when we talk about serving benchmark, do we only care about the inference performance of a ML model, or do we care about the whole serving stack (that would include caching, networking, scaling, etc.), or something in between?
- what are the metrics do we care about and what impacts those metrics? i assume there is throughput party and latency party, but for each case, what factors shall we take into consideration other than the ML model itself? e.g. a caching layer would be important for throughput, hence we might want to consider these when performing benchmarks.
- any suggestions or insights from real world use cases?
jbottum
chrisheecho
What's the plan for serving benchmark?
We need to
any thought?