Some questions about the code

[ThoughtsAreStarry]

Hello, thanks for your excellent work! There are some questions about the code, looking forward to your reply. Q1.

def _lookup(self, x, table_q, scale):
        if self.training:
            grid = (x / scale).clamp(-1, 1)
            if self.is_act:
                wgt = torch.histc(grid.data, table_q.numel() // 2 + 1).float().view(1, 1, -1, 1).sqrt()
                wgt = F.pad(wgt, [0, 0, table_q.numel() // 2, 0]) + 1e-5
                table_q = table_q.data + (table_q - table_q.data) / wgt * x.numel() / (table_q.numel() // 2 + 1)        
            else:
                wgt = torch.histc(grid.data, table_q.numel()).float().view(table_q.shape).sqrt() + 1e-5
                table_q = table_q.data + (table_q - table_q.data) / wgt * x.numel() / table_q.numel()   

In this code, the update of table_q is invalid, is there a problem here? Q2.

def forward(self, x):
        if bool(self.scale == 0):
            if self.is_act:
                self.scale.data = (x.std() * 3).log()
            else:
                self.scale.data = (x.std() * 3).log()
        scale = self.scale.exp()                             
        if self.training:
            # generate lookup table
            table_q = self._gen_table()

            # lookup operation
            x_q = self._lookup(x, table_q, scale)
        else:
            # lookup operation
            x_q = self._lookup(x, self.table_q, scale)
        return x_q

During model inference, the scale needs to be calculated from the statistical values of the input. Did you try to use a fixed trained scale at inference time.

[LongguangWang]

Thanks for your interests in our work. Q1: table_q = table_q.data + (table_q - table_q.data) / wgt * x.numel() / (table_q.numel() // 2 + 1) still preserves the gradients of table_q and only rescale the gradient values for different bins in the table. Therefore, table_q can still be updated during training. Q2: self.scale.data = (x.std() * 3).log() is only used to initialize the scale paramter at the first iteration and will then be updated by the gradient during training. During inference, learned scale paramter will be adopted.

[ThoughtsAreStarry]

Thanks for your reply. I've noticed a misunderstanding I've had about gradient propagatio. I'm used to rewriting backward for backpropagation, ignoring that using tensor and tensor.data (required_grad=False in tensor.data ) can also achieve the rescaling of gradient. Besides, I have a question about the hyperparameter self.granu. As mentioned in the paper, With larger granularity, the degree of freedom for our lookup tables is increased such that better performance is achieved. Since granularity larger than 9 does not provide further notable improvement, K is set to 9 as the default setting. This conclusion based on experiments with 4-bits or less, whether K should be larger in the case of higher bits (6-bits to 9-bits)? Thanks again.

[Feynman1999]

Hello, thanks for your excellent work! There are some questions about the code, looking forward to your reply. Q1.

def _lookup(self, x, table_q, scale):
        if self.training:
            grid = (x / scale).clamp(-1, 1)
            if self.is_act:
                wgt = torch.histc(grid.data, table_q.numel() // 2 + 1).float().view(1, 1, -1, 1).sqrt()
                wgt = F.pad(wgt, [0, 0, table_q.numel() // 2, 0]) + 1e-5
                table_q = table_q.data + (table_q - table_q.data) / wgt * x.numel() / (table_q.numel() // 2 + 1)        
            else:
                wgt = torch.histc(grid.data, table_q.numel()).float().view(table_q.shape).sqrt() + 1e-5
                table_q = table_q.data + (table_q - table_q.data) / wgt * x.numel() / table_q.numel()   

In this code, the update of table_q is invalid, is there a problem here? Q2.

def forward(self, x):
        if bool(self.scale == 0):
            if self.is_act:
                self.scale.data = (x.std() * 3).log()
            else:
                self.scale.data = (x.std() * 3).log()
        scale = self.scale.exp()                             
        if self.training:
            # generate lookup table
            table_q = self._gen_table()

            # lookup operation
            x_q = self._lookup(x, table_q, scale)
        else:
            # lookup operation
            x_q = self._lookup(x, self.table_q, scale)
        return x_q

During model inference, the scale needs to be calculated from the statistical values of the input. Did you try to use a fixed trained scale at inference time.

i do not understand why use the '.sqrt()' function, can you explain it? thanks !