[HotCloud'19] Accelerating Deep Learning Inference via Freezing

[ganler]

https://www.usenix.org/conference/hotcloud19/presentation/kumar

[ganler]

I like this summary:

[ganler]

Computer system:

Computer Data => Spatial Locality => Cache. Video Frames => Temporal Locality => Inference Cache?

Exact cache hit is unlikely

=> Approximate Caching?

1. Compare the internal output with outputs in the cache. Rank the K-nearest neighbors. Prediction can be more confident if:
   • More neighbors agree on the same label.
   • Neighbors are closer to the input point.

Parameter: for each layer: a threshold to tell YES or NO.

During inference, a cache lookup is done after every layer and a cache hit yields a faster prediction.

[ganler]

The result looks good. (Too good to be true?)