QAT references

[volcacius]

Papers:

• LSQ+: https://arxiv.org/abs/2004.09576
• Rotated binary network: https://arxiv.org/abs/2009.13055
• PROFIT: https://link.springer.com/chapter/10.1007/978-3-030-58539-6_26
• Exponential rescaling: https://arxiv.org/abs/1808.05779
• Soft rounding differentiable approximation: https://arxiv.org/pdf/2006.09952.pdf and https://arxiv.org/abs/1908.05033
• Scaled elementwise STE https://arxiv.org/abs/2104.00903
• https://openaccess.thecvf.com/content_CVPRW_2020/papers/w40/Yu_Low-Bit_Quantization_Needs_Good_Distribution_CVPRW_2020_paper.pdf

[volcacius]

Soft rounding:

#Fwd
def soft_round(x, alpha):
    x_floor = torch.floor(x)
    r = x - x_floor - 0.5
    out = x_floor + 0.5 * torch.tanh(alpha * r) / torch.tanh(alpha * 0.5) + 0.5
    return out