Open andeyeluguo opened 6 years ago
Hi @andeyeluguo,
It's a linear regression problem, and minimization function is least square error. So OLS would be fine, but I use the scipy.linalg.lstsq lib instead.
In fact, it's NOT an efficient way to approximate the linear combination, because the linear regression problem has few parameters(1~5 unknown alpha) but large amount of quantized samples(256965*5 first bin layer). So I think author maybe avoid this problem through some other method.
As for using batch norm to replace the alpha, I don't think that batch norm can combine the effect of each base to approximate F.P. weights.
@andeyeluguo @cow8 Hello, I use the batch norm as beta to approximate activations , now its training; do you have any good idea for activations.
I recommend using forward method which means replacing the original ops with quantized ops in a cumulative manner. And testing if it converge when trained without activation-Quantized may be a good idea @niluanwudidadi
@cow8 Thank you for your reply, I will have a try!
I am confused by the alpha value too。I try to use batch norm to replace the alpha,but it did't work。