search

huawei-noah / AdderNet

Code for paper " AdderNet: Do We Really Need Multiplications in Deep Learning?"
BSD 3-Clause "New" or "Revised" License
952 stars 187 forks source link

the speed of adder.adder2d slower than torch.nn.Conv2d ? I'm confused. #13

Closed zccyman closed 3 years ago

zccyman commented 4 years ago

I replace adder.adder2d with torch.nn.Conv2d, and replace torch.cdist with my_cdist. train my network, the speed of the new model slower than the old at least 6 times. I'm confused.

@torch.jit.script
def my_cdist(x1, x2):
    x1_norm = x1.pow(2).sum(dim=-1, keepdim=True)
    x2_norm = x2.pow(2).sum(dim=-1, keepdim=True)
    res = torch.addmm(x2_norm.transpose(-2, -1), x1, x2.transpose(-2, -1), alpha=-2).add_(x1_norm)
    res = res.clamp_min_(1e-30).sqrt_()

    return res
HantingChen commented 4 years ago

nn.conv2d is acclerated by using cuda and cudnn, which is defintely faster than the adder2d without cuda accleration.

zccyman commented 4 years ago

@HantingChen Thanks! I want to know how many times the adder2d with cuda accleration faster than nn.conv2d ?

HantingChen commented 4 years ago

@HantingChen Thanks! I want to know how many times the adder2d with cuda accleration faster than nn.conv2d ?

Since the speed of 32bit multiplications and additions in GPU are almost the same, adder and conv will take almost the same time if with cuda accleration

buttercutter commented 4 years ago

@zccyman Do you have idea about using YOLOv3 together with AdderNet ?