Custom Layer Normalization Metal Kernel

liuliu / ccv

C-based/Cached/Core Computer Vision Library, A Modern Computer Vision Library

http://libccv.org

Other

7.07k stars 1.71k forks source link

Closed philipturner closed 1 year ago

philipturner commented 1 year ago

IO-aware exact and approximate normalization. Only applied to layer normalization right now.

Archived draft with the full FlashNorm implementation:

Gathered mean and standard deviation of a subset of the population, as large as could fit into SRAM (16,000 elements).
Random number generator sourced from MolecularRenderer. The generator was originally designed for ray tracing.