forward() suspend when training

boykac commented 4 years ago

训练时resnet():forward()中前面可以正常运算，到x4=self.layer3(x3)就停住了。GPU里放着模型，但一点运算都没有

boykac commented 4 years ago

def forward(self, x):
    x = self.relu1(self.bn1(self.conv1(x)))
    x = self.relu2(self.bn2(self.conv2(x)))
    x = self.relu3(self.bn3(self.conv3(x)))
    x = self.maxpool(x)
    x2 = self.layer1(x)
    x3 = self.layer2(x2)
    # 1-这里可以正常输出x3
    x4 = self.layer3(x3)
    # no-这里就什么都没有，程序就卡在这里了，GPU也没有运算
    x5 = self.layer4(x4)
    x = self.layer5(x5)
    edge,edge_fea = self.edge_layer(x2,x3,x4)
    seg1,x = self.layer6(x,x2)
    x = torch.cat([x, edge_fea], dim=1)
    seg2 = self.layer7(x)

    return [[seg1, seg2], [edge]]

boykac commented 4 years ago

但是在evaluation的时候没有问题

liutinglt commented 4 years ago

@boykac Have you try smaller batch size?

liutinglt / CE2P

forward() suspend when training #38