This PR adds a new operator and two replacement passes that will be useful for deploying llama-based transformer models.
Added
A new PACT operator for the RMSNormalization operation from this paper. As PyTorch doesn't have an RMSNorm module yet, users need to provide a custom trace and module to match with.
A new replacement pass in approximate.py to approximate SiLU modules into PACTGelu.
This PR adds a new operator and two replacement passes that will be useful for deploying llama-based transformer models.
Added
RMSNormalizationoperation from this paper. As PyTorch doesn't have an RMSNorm module yet, users need to provide a custom trace and module to match with.approximate.pyto approximate SiLU modules into PACTGelu.