Why do you transform confidence through affinity instead of using confidence directly?

[Erik-Y]

Hi, I'm very curious why do you transform confidence through affinity instead of using confidence directly in the following code.

------from nlspnmodel.py 115 ------

if self.args.conf_prop: list_conf = [] offset_each = torch.chunk(offset, self.num + 1, dim=1)

        modulation_dummy = torch.ones((B, 1, H, W)).type_as(offset).detach()

        for idx_off in range(0, self.num + 1):
            ww = idx_off % self.k_f
            hh = idx_off // self.k_f

            if ww == (self.k_f - 1) / 2 and hh == (self.k_f - 1) / 2:
                continue

            offset_tmp = offset_each[idx_off].detach()
            offset_tmp[:, 0, :, :] = \
                offset_tmp[:, 0, :, :] + hh - (self.k_f - 1) / 2
            offset_tmp[:, 1, :, :] = \
                offset_tmp[:, 1, :, :] + ww - (self.k_f - 1) / 2

            conf_tmp = ModulatedDeformConvFunction.apply(
                confidence, offset_tmp, modulation_dummy, self.w_conf,
                self.b, self.stride, 0, self.dilation, self.groups,
                self.deformable_groups, self.im2col_step)
            list_conf.append(conf_tmp)

        conf_aff = torch.cat(list_conf, dim=1)
        aff = aff * conf_aff.contiguous()

[zzangjinsun]

Hello @Erik-Y,

The variable aff is the affinity values for non-local neighbors with sub-pixel offsets. The proposed confidence-incorporated affinity normalization adjusts those affinities based on each non-local neighbor's confidence.

Thus, we need to calculate each non-local neighbor's confidence, which should be also sampled from the non-local neighbor's offset location in the confidence map.

[zerowfz]

hi, i am also curious about this part. I have known that we should calculate each non-local neighbor's confidence,.But ,why the offset for confidence is changed by

offset_tmp[:,0,:,:] = offset_tmp[:, 0, :, :] + hh - (self.k_f - 1) / 2

why the offset for confidence is not same with the offset for propagation process???

[Erik-Y]

Hi, @zzangjinsun I understand the "The variable aff is the affinity values for non-local neighbors with sub-pixel offsets. The proposed confidence-incorporated affinity normalization adjusts those affinities based on each non-local neighbor's confidence.Thus, we need to calculate each non-local neighbor's confidence", but I don not understand "which should be also sampled from the non-local neighbor's offset location in the confidence map"

In my opinion, for every location (x,y) in (H,W), original confidence and original aff have corresponding values. Thus, its more reasonable to use the original confidence in confidence-incorporated affinity normalization. Can you give me more specific theories or some experiment results to help me? Thanks a lot.

[zzangjinsun]

Thank you for the important discussions on confidence, @Erik-Y , @zerowfz . Your intuition is right and I need to thoroughly check this part again after the CVPR period.

Please give me some time to check again!!

[Erik-Y]

@zzangjinsun Of course. Good luck

[zzangjinsun]

I fixed and updated the repo.

Thank you again @Erik-Y and @zerowfz for your intuitions.