Minor bug in model quantization - Githubissues

heheda365 / ultra_net

FPGA-based neural network inference project for 2020 DAC System Design Contest

109 stars 40 forks source link

Minor bug in model quantization #9

Open fffasttime opened 1 year ago

fffasttime commented 1 year ago

https://github.com/heheda365/ultra_net/blob/dc3b339da7f7e27276414a5c4dd7ecf89f7fc605/quantization/quant_ultra.py#L173-L174 每层计算有0.3%左右的误差

对照torch文档 https://pytorch.org/docs/1.3.0/nn.html，此处eps应该在sqrt内部