ADLS: Hardware Normalization (Group 17)

[dereklai1]

Hardware Normalization (WIP)

Note: There are a bunch of unrelated changes mainly in mase_components/matmul/ (duplicate of #53) due to our rebase onto the HEAD of the private repo mase-tools. Every relevant change to the coursework is detailed below.

Software:

• Added BatchNorm2d, LayerNorm, GroupNorm, InstanceNorm2d, and RMSNorm into QUANTIZABLE_OP.
• Added RMSNorm layer
• Added input integer quantized layers BatchNorm2dInteger, LayerNormInteger, GroupNormInteger, InstanceNorm2dInteger, RMSNormInteger.
• Changed all metadata passes to support auto-generation of integer/fixed-point quantized normalization hardware from MASE graph.
• Refactored emit_internal, emit_logicnets, etc.. to delete redundant INTERNAL_RTL_DEPENDENCIES dictionary and replaced it with INTERNAL_COMP.
• Replaced emit_tb pass to use string templating instead of dill'ing a testbench class.
• Added C++ model/simulation of the Newton-Raphson method to calculate the inverse square root of a fixed-point number.
• Added test_emit_verilog_norm.py which tests all types of normalization layer hardware generation starting from PyTorch NNs. (RMSNorm currently not working due to circular import problem)

Hardware:

• Added fixed_isqrt which uses Newton-Raphson iteration to compute inverse square roots.
• Added batch_norm_2d which implements batch normalization including affine transformations.
• Added group_norm_2d which can be specialized to implement group norm, layer norm, and instance norm and instance norm without any affine transforms.
• Added single entry point module norm for all normalization layers. The type of normalization can be chosen through parameters.
• All hardware modules listed above have corresponding cocotb tests which are all passing.
  • Inverse square root test sweeps all possible values within a bit width.
  • All other tests are run with fully random input, output back-pressure and input valid patterns.
• Added new ErrorThresholdStreamMonitor stream monitor which checks that the output value is within a certain threshold of last-bit errors.
• Added norm_synth_impl.tcl Vivado script and alveo-u250-norm.xdc to run synthesis and implementation automatically and generate resource usage and timing reports for various parameterizations.
• Added standalone simulate_pass() into mase_cocotb.runner to support running MASE-generated hardware projects.
• Created bit-accuracy hardware behavior models for group_norm_2d and fixed_isqrt which can be used to check hardware errors. Note that the quantized software layers are used in the respective layers and not the hardware behavior models.
• Added sphinx_docs for norm modules.

Minor Changes:

• Added matrix_fifo which combines matrix_flatten, fifo and matrix_unflatten.
• Fixed issues with fifo and repeat_circular_buffer.
• Refactored fixed_signed_cast_tb to extract out software model.
• Fixed bugs in mase_cocotb.monitor and mase_cocotb.driver.
• Improved the parameter print of mase_cocotb.runner and fixed compile options for Verilator.
• Added more utility functions into mase_cocotb.utils.
• Added integer floor quantizer into quantizers.integer.

Further Work:

• MASE CI
• [MASE bug] There's still a 1-cycle output throughput drop in fixed_accumulator module.
• Circular import needs to be fixed for RMSNorm integration in metadata passes.
• Add affine transforms for layer norm, instance norm, group norm, batch norm.

[jianyicheng]

@dereklai1

Hi, we found there is a bug the yml files that stop CI running from forked repos. It has now been fixed and merged into the upstream. Could you merge the main branch of the upstream into this PR which should trigger the CI properly?

Thanks,