Closed pkhdipraja closed 2 months ago
Hey @dvdblk, Thank you for your time and taking a look at the pull request of @pkhdipraja !
You're right that he could use the LinearEpsilon
instead of nn.Linear
. But there is also another way to do it: we could keep all nn.Linear
and instead apply the lxt.rules.EpsilonRule
on all nn.Linear
layers in the Composite
. If you look at line 63 in bert.py, you see that @pkhdipraja did apply this rule on all nn.Linear
.
At the end, it depends on your taste but both approaches are correct. Actually, I benchmarked once both approaches and saw that the way of @pkhdipraja is faster! I should mention this in the documentaion.
Best greetings!
@dvdblk nice, you already edited your comment! (:
Hey @pkhdipraja ,
Thank you so much for adding another model to this library! The code works as expected in my tests! I pull it into the main branch.
Best
Isn't it missing the use of
LinearEpsilon
instead ofLinear
? Also inprune_linear_layer
.EDIT: nvm this is handled by the Composite :)