Espnetv2 redundant avg pooling in input reinforcement

In the input reinforcement branches it seem that there are redundants Average pool on the original input in the deeper stages. Do you see any problem to pass the strided average input image after each DownSample block to the next one? In DownSampler forward method, return input2:

def forward(self, input, input2=None):
        '''
        :param input: input feature map
        :return: feature map down-sampled by a factor of 2
        '''
        avg_out = self.avg(input)
        eesp_out = self.eesp(input)
        output = torch.cat([avg_out, eesp_out], 1)

        if input2 is not None:
            #assuming the input is a square image
            # Shortcut connection with the input image
            w1 = avg_out.size(2)
            while True:
                input2 = F.avg_pool2d(input2, kernel_size=3, padding=1, stride=2)
                w2 = input2.size(2)
                if w2 == w1:
                    break
            output = output + self.inp_reinf(input2)

        return self.act(output), input2

In Espnet class, overwrite the input object:

out_l3_0, input = self.level3_0(out_l2, input)  # down-sample

sacmehta / EdgeNets

Espnetv2 redundant avg pooling in input reinforcement #40