Closed yan-ctrl closed 1 year ago
Hi @yan-ctrl, thanks for your questions. For questions 1&2: the answer is yes. The gradient is collected at the input of the loss function (or the output of the last layer of the model, they are equivalent). All models follow this rule including the AlphAction. For question 3, since many issues ask me for a manual formula of the gradient calculation, I can provide some pseudocode here. Note that the manual calculation and the hook calculation are mutually exclusive, you can only choose one to use.
def collect_grad_manual(self, input, target, focus_factor, normalizer):
prob = torch.sigmoid(input)
pos_grad = target * torch.pow((1 - prob), focus_factor) * (focus_factor * prob * torch.log(prob) + prob - 1) * focus_factor / self.focal_gamma
neg_grad = (target - 1) * torch.pow(prob, focus_factor) * (focus_factor * (1 - prob) * torch.log(1 - prob) - prob) * focus_factor / self.focal_gamma
if self.focal_alpha >= 0:
pos_grad = pos_grad * self.focal_alpha
neg_grad = neg_grad * (1 - self.focal_alpha)
grad = pos_grad + neg_grad
grad = torch.abs(grad / normalizer)
pos_grad = torch.sum(grad * target, dim=0)
neg_grad = torch.sum(grad * (1 - target), dim=0)
allreduce(pos_grad)
allreduce(neg_grad)
self.pos_grad += pos_grad
self.neg_grad += neg_grad
self.pos_neg = torch.clamp(self.pos_grad / (self.neg_grad + 1e-10), min=0, max=1)
For question 4, the softmax loss function only calculates at the one-hot of 1, but the gradient is backward to all classes because different categories are competitive in softmax. Therefore, the definition of positive and negative gradients still holds. And the softmax version of the equalization loss could be found here.
嗨@yan-ctrl,感谢您的提问。对于问题1和2:答案是肯定的。梯度是在损失函数的输入(或模型最后一层的输出,它们是等效的)处收集的。所有模型都遵循此规则,包括AlphAction。对于问题 3,由于许多问题要求我手动提供梯度计算公式,我可以在这里提供一些伪代码。请注意,手动计算和钩子计算是互斥的,您只能选择一个使用。
def collect_grad_manual(self, input, target, focus_factor, normalizer): prob = torch.sigmoid(input) pos_grad = target * torch.pow((1 - prob), focus_factor) * (focus_factor * prob * torch.log(prob) + prob - 1) * focus_factor / self.focal_gamma neg_grad = (target - 1) * torch.pow(prob, focus_factor) * (focus_factor * (1 - prob) * torch.log(1 - prob) - prob) * focus_factor / self.focal_gamma if self.focal_alpha >= 0: pos_grad = pos_grad * self.focal_alpha neg_grad = neg_grad * (1 - self.focal_alpha) grad = pos_grad + neg_grad grad = torch.abs(grad / normalizer) pos_grad = torch.sum(grad * target, dim=0) neg_grad = torch.sum(grad * (1 - target), dim=0) allreduce(pos_grad) allreduce(neg_grad) self.pos_grad += pos_grad self.neg_grad += neg_grad self.pos_neg = torch.clamp(self.pos_grad / (self.neg_grad + 1e-10), min=0, max=1)
对于问题 4,softmax 损失函数仅在 1 中的 one-hot 处计算,但梯度向后到所有类别,因为不同的类别在 softmax 中具有竞争力。因此,正梯度和负梯度的定义仍然成立。均衡损耗的softmax版本可以在这里找到。 Thank you very much for your patient answer, which is very helpful to me. The pseudocode is very clear. I will try to apply it to my own tasks. I have also read the paper you recommended, so I have another question to ask you, for the Focusing Factor of Softmax-EFL, whether Gj only uses the cumulative gradient of positive samples of each category, and whether it is necessary to substitute the cumulative gradient of positive samples into the softmax formula to recalculate the prediction probability pj like Softmax-EQL?
Thank you very much for your patient answer, which is very helpful to me. The pseudocode is very clear. I will try to apply it to my own tasks. I have also read the paper you recommended, so I have another question to ask you, for the Focusing Factor of Softmax-EFL, whether Gj only uses the cumulative gradient of positive samples of each category, and whether it is necessary to substitute the cumulative gradient of positive samples into the softmax formula to recalculate the prediction probability pj like Softmax-EQL?
@yan-ctrl, EFL (Equalized Focal Loss) is a generalized version of the sigmoid focal loss. We have not yet proposed a softmax version EFL, and you could try it by yourself. Meanwhile, I have a little tip could give you: If the rank between categories is crucial in your task (like image classification task must choose the argmax prediction of an instance), applying a softmax-like loss function with the cumulative gradient of positive samples or negative samples is better. Otherwise (like object detection task calculate the mAP of each category independently), you could choose a sigmoid-like loss function and use the gradient ratio as the long-tail indicator.
Thank you for your patience. I see
Hello, when I apply sigmoid-EFL to yolov5, I replace the classification loss of yolov5 with sigmoid-EFL, the training batch Size is 32, lr is 0.01, single card training, RTX3080, set super parameter fl The gamma is 1.5, and the super parameter fl is added_ Alpha=0.25. My code is shown below. However, when training to the second epoch, the classification loss appears a nan value. As shown below:
Training Log: Epoch gpu_mem box obj cls labels img_size 0/299 6.45G 0.05136 0.01066 0.004802 430 640: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.02it/s] val: Scanning '../datasets/coco128/labels/train2017.cache' images and labels... 128 found, 0 missing, 2 empty, 0 corrupted: 100%|██████████████████████████████████████████████████████████████████████| 128/128 [00:10<?, ?it/s] Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:04<00:00, 2.43s/it] all 128 929 0.662 0.157 0.189 0.0993
Epoch gpu_mem box obj cls labels img_size
1/299 8.08G 0.04997 0.009346 0.003714 468 640: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.17it/s]
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:12<00:00, 6.18s/it]
all 128 929 0.454 0.151 0.141 0.0736
Epoch gpu_mem box obj cls labels img_size
2/299 8.08G 0.04928 0.008183 0.003707 404 640: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.14it/s]
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:12<00:00, 6.29s/it]
all 128 929 0.498 0.143 0.155 0.0756
Epoch gpu_mem box obj cls labels img_size
3/299 8.08G 0.04829 0.009446 nan 440 640: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.26it/s]
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:11<00:00, 5.54s/it]
all 128 929 0.472 0.145 0.157 0.0783
Epoch gpu_mem box obj cls labels img_size
4/299 8.08G 0.04913 0.009771 nan 541 640: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 3.30it/s]
Class Images Labels P R mAP@.5 mAP@.5:.95: 100%|███████████████████████████████
I have tried to turn the batch down Size and learning rate, but it has no effect. We also tried to add a small value of 1e-10 in the torch. log() operation and torch. pow() operation, but it still has no effect. When I use torch.autograd.set detect Anonymous (True) and with torch.autograd.detect anomaly():
When scaler. scale (loss). backward() is placed at the beginning of the training code and backpropagation, the following errors are reported respectively: 1、[W python_anomaly_mode.cpp:104] Warning: Error detected in MulBackward0. Traceback of forward call that caused the error: File "/root/yolov5-6.0/utils/loss.py", line 157, in forward efl = ce_loss torch.pow((1-pred_t), ff.detach()) wf.detach() (function _print_stack) 3/299 8.08G 0.04854 0.01028 0.003171 448 640: 25%|██
2、File "/root/miniconda3/lib/python3.8/site-packages/torch/autograd/init.py", line 145, in backward Variable._execution_engine.run_backward( RuntimeError: Function 'MulBackward0' returned nan values in its 0th output.
I haven't been able to solve it. Could you help me to see what problems exist in the code, or give me some suggestions.
class sigmoidEFL(nn.Module): def init(self, loss_fcn, focal_gamma_b=1.5, focal_alpha=0.25, loss_weight=1.0, ignore_index=-1, num_classes=80, scale_factor=8.0, indicator='pos_and_neg'): super(sigmoidEFL,self).init() self.loss_fcn = loss_fcn self.reduction = loss_fcn.reduction self.loss_fcn.reduction = 'none' self.num_classes = num_classes self.loss_weight = loss_weight self.ingnore_index = ignore_index #
self.focal_gamma = focal_gamma_b
self.focal_alpha = focal_alpha
# # ignore bg class and ignore idx
# self.num_classes = num_classes - 1
# cfg for efl loss
self.scale_factor = scale_factor
assert indicator in ['pos', 'neg', 'pos_and_neg'], 'Wrong indicator type!'
self.indicator = indicator
# initial variables
self.register_buffer('pos_grad', torch.zeros(self.num_classes).to(device_efl))
self.register_buffer('neg_grad', torch.zeros(self.num_classes).to(device_efl))
self.register_buffer('pos_neg', torch.ones(self.num_classes).to(device_efl))
# grad collect
# self.grad_buffer = [] # don't understand
def forward(self, logits, label): #normalizer=None
if self.indicator == 'pos':
indicator = self.pos_grad.detach()
# indicator = self.grad_buffer[0]
elif self.indicator == 'neg':
indicator = self.neg_grad.detach()
elif self.indicator == 'pos_and_neg':
indicator = self.pos_neg.detach()
# indicator = self.pos_neg.detach() + self.neg_grad.detach()
else:
raise NotImplementedError
self.n_c = logits.shape[-1]
self.logits = logits.reshape(-1, self.n_c)
self.n_i, _ = self.logits.size()
#one-hot
def expand_label(pred, gt_classes):
target = pred.new_zeros(self.n_i, self.n_c)
target[torch.arange(self.n_i), gt_classes] = 1
return target
if label.dim() == 1:
expand_target = expand_label(self.logits,label)
else:
expand_target = label.clone()
self.targets = expand_target
pred = torch.sigmoid(logits)
pred_t = pred * self.targets + (1 - pred) * (1 - self.targets)
alpha = torch.tensor(self.focal_alpha)
# indicator = torch.clamp(indicator,min=0,max=1) # if indicator == 'pos' or neg
map_val = 1 - indicator
dy_gamma = self.focal_gamma + self.scale_factor * map_val
# focusing factor
ff = dy_gamma.view(1, -1).expand(self.n_i, self.n_c)
# weighting factor
wf = ff / self.focal_gamma
# ce_loss = F.binary_cross_entropy_with_logits(self.logits,self.targets,reduction='none') #
ce_loss = self.loss_fcn(self.logits, self.targets)
efl = ce_loss * torch.pow((1 - pred_t), ff.detach()) * wf.detach()
# to avoid an OOM error
# torch.cuda.empty_cache()
if self.focal_alpha >= 0:
alpha_t = self.focal_alpha * self.targets + (1 - self.focal_alpha) * (1 - self.targets)
efl = alpha_t * efl
self.collect_grad(self.logits.detach(), self.targets.detach(),ff.detach(),wf.detach(),alpha.detach())
if self.reduction == 'mean':
return efl.mean()
elif self.reduction == 'sum':
return efl.sum()
else: # 'none'
return efl
# self.collect_grad(self.inputs, targets, self.outputs)
def collect_grad(self, logits, targets,ff,wf,alpha):
pred = torch.sigmoid(logits)
pos_grad = targets * torch.pow((1-pred),ff) * (ff * pred * torch.log(pred+1e-10) + (pred-1)) * wf
neg_grad = (1-targets) * torch.pow(pred,ff) * (ff * (1-pred) * torch.log(1-pred+1e-10) - pred) * wf
if alpha >= 0:
pos_grad = pos_grad * alpha
neg_grad = neg_grad * (1 - alpha)
grad = pos_grad + neg_grad
grad = torch.abs(grad)
pos_grad = torch.sum(grad * targets,dim=0)
neg_grad = torch.sum(grad * (1-targets),dim=0)
# allreduce(pos_grad)
# allreduce(neg_grad)
self.pos_grad += pos_grad
self.neg_grad += neg_grad
self.pos_neg = torch.clamp(self.pos_grad / (self.neg_grad + 1e-10), min=0, max=1)
Hi @yan-ctrl, Nan problem may be caused by many reasons. You could:
Thank you for your suggestion. There are some problems with focal loss
Thank you for your suggestion. There are some problems with focal loss
老哥,请问你在yolov5上应用Softmax-EFL是否有效?我也出现了Nan的问题,请问您是否已解决?
Hello, I have also encountered a problem with my own EFL, which is to calculate the gradient manually. After reading your answers to several questions, you suggest using HOOK to collect the gradient, and I would like to ask you a few questions: