search

NoamRosenberg / autodeeplab

AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch
MIT License
308 stars 73 forks source link

A mistake about the network level parameters! #65

Closed huizhang0110 closed 5 years ago

huizhang0110 commented 5 years ago

In the origin paper, the network level update is

Screen Shot 2019-10-18 at 3 35 25 PM

But in your code:

           # normalized_betas[layer][ith node][0 : ➚, 1: ➙, 2 : ➘]
            for layer in range (len(self.betas)):
                if layer == 0:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)

                elif layer == 1:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)

                elif layer == 2:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)
                    normalized_betas[layer][2] = F.softmax (self.betas[layer][2].to(device=img_device), dim=-1)
                else :
                    `normalized_betas[layer][0][1:]` = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)
                    normalized_betas[layer][2] = F.softmax (self.betas[layer][2].to(device=img_device), dim=-1)
                    normalized_betas[layer][3][:1] = F.softmax (self.betas[layer][3][:1].to(device=img_device), dim=-1)
HankKung commented 5 years ago

In the origin paper, the network level update is

Screen Shot 2019-10-18 at 3 35 25 PM

But in your code:

           # normalized_betas[layer][ith node][0 : ➚, 1: ➙, 2 : ➘]
            for layer in range (len(self.betas)):
                if layer == 0:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)

                elif layer == 1:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)

                elif layer == 2:
                    normalized_betas[layer][0][1:] = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)
                    normalized_betas[layer][2] = F.softmax (self.betas[layer][2].to(device=img_device), dim=-1)
                else :
                    `normalized_betas[layer][0][1:]` = F.softmax (self.betas[layer][0][1:].to(device=img_device), dim=-1)
                    normalized_betas[layer][1] = F.softmax (self.betas[layer][1].to(device=img_device), dim=-1)
                    normalized_betas[layer][2] = F.softmax (self.betas[layer][2].to(device=img_device), dim=-1)
                    normalized_betas[layer][3][:1] = F.softmax (self.betas[layer][3][:1].to(device=img_device), dim=-1)

This section of code is the implementation of beta normalization before further inference like the formula you mentioned. image