how do you use the lstsq to get the alpha value？

[andeyeluguo]

I am confused by the alpha value too。I try to use batch norm to replace the alpha，but it did't work。

[zhuyinheng]

Hi @andeyeluguo,

It's a linear regression problem, and minimization function is least square error. So OLS would be fine, but I use the scipy.linalg.lstsq lib instead.

In fact, it's NOT an efficient way to approximate the linear combination, because the linear regression problem has few parameters(1~5 unknown alpha) but large amount of quantized samples(256965*5 first bin layer). So I think author maybe avoid this problem through some other method.

As for using batch norm to replace the alpha, I don't think that batch norm can combine the effect of each base to approximate F.P. weights.

[niluanwudidadi]

@andeyeluguo @cow8 Hello, I use the batch norm as beta to approximate activations , now its training; do you have any good idea for activations.

[zhuyinheng]

I recommend using forward method which means replacing the original ops with quantized ops in a cumulative manner. And testing if it converge when trained without activation-Quantized may be a good idea @niluanwudidadi

[niluanwudidadi]

@cow8 Thank you for your reply， I will have a try！