preds = self.net_g(self.lq)
if not isinstance(preds, list):
preds = [preds]
self.output = preds[-1]
loss_dict = OrderedDict()
# pixel loss
l_pix = 0.
for pred in preds:
l_pix += self.cri_pix(pred, self.gt)
if math.isnan(l_pix):
print(self.name)
time.sleep(1000000000000)
loss_dict['l_pix'] = l_pix
l_pix.backward()
"""
for name, param in self.net_g.named_parameters():
if param.grad is None:
print(name)
"""
if self.opt['train']['use_grad_clip']:
torch.nn.utils.clip_grad_norm_(self.net_g.parameters(), 0.01)
self.optimizer_g.step()
self.log_dict = self.reduce_loss_dict(loss_dict)
if self.ema_decay > 0:
self.model_ema(decay=self.ema_decay)`
When I was training the I-HAZE dataset, I found that the output of the model was nan.
basicsr.models.image_restoration_model.py
`def optimize_parameters(self, current_iter): self.optimizer_g.zero_grad()
self.name = ./data/IH-HAZE/train/hazy/IH_13_indoor_hazy.jpg
It's always this image，but this situation did not happen in the first Epoch.