This PR adds support for Layer Normalization using either Keras or PyTorch with the Vivado backend in io_parallel mode.
This implementation uses a lookup table for inverse square root; the inputs to the lookup table follow a logarithmic distribution for better accuracy.
Tests have been added for both Keras and Pytorch parsing.
Credit is due to @Ethan0Jiang and @LostEcho365 (Zhixing Jiang and Dennis Yin) for their Vivado implementation and Keras parsing support; my contributions were making a change to the inverse square root lookup table implementation, implementing PyTorch support, and adding unit tests. (Here's a link to their pre-print.) The original code authors have given permission for their code to be merged into hls4ml.
The pytest failure is the QKeras tests timing out. There's 299 tests being run in that batch, which I guess is too many. Is there a why to reshuffle the batches to avoid the timeout?
Description
This PR adds support for Layer Normalization using either Keras or PyTorch with the Vivado backend in
io_parallel
mode.This implementation uses a lookup table for inverse square root; the inputs to the lookup table follow a logarithmic distribution for better accuracy.
Tests have been added for both Keras and Pytorch parsing.
Credit is due to @Ethan0Jiang and @LostEcho365 (Zhixing Jiang and Dennis Yin) for their Vivado implementation and Keras parsing support; my contributions were making a change to the inverse square root lookup table implementation, implementing PyTorch support, and adding unit tests. (Here's a link to their pre-print.) The original code authors have given permission for their code to be merged into hls4ml.
Linked issue: https://github.com/fastmachinelearning/hls4ml/issues/1109
Type of change
Tests
Two unit tests added:
test/pytest/test_layernorm.py
andtest/pytest/test_layernorm_pytorch.py
Checklist
pre-commit
on the files I edited or added.