search

yu4u / noise2noise

An unofficial and partial Keras implementation of "Noise2Noise: Learning Image Restoration without Clean Data"
MIT License
1.07k stars 235 forks source link

Loss is extreamly small from the very beginning. #8

Open kawori opened 6 years ago

kawori commented 6 years ago

I found that when training with an image with different noises, the MSE loss is very small from the beginning and then underflow.

kawori commented 6 years ago

What causes this issue is that I normalized the images to the range [0, 1].

yu4u commented 6 years ago

Is it solved?

kawori commented 6 years ago

Solved. I'm using UNet for training, but it's hard to converge. It's the same for RedNet.

kawori commented 6 years ago

When using UNet, the loss decresed fast at first, but then decreased slowly and increased to INF suddenly.

kawori commented 6 years ago

I've create a repo here. You can review my codes though it's PyTorch implementation.

yu4u commented 6 years ago

I added UNet to this repository, and trained it. It works and seems to be better than SRResNet in case of Gaussian noise. I think the problem is not due to the UNet model.

kawori commented 6 years ago

My UNet model has no BN and dropout layer. Its detailed parameters are from Noise2Noise paper's appendix.

yu4u commented 6 years ago

Even if the model is trained with no BN and dropout, it works in my implementation. This can be done by calling get_unet_model with dropout=0.

https://github.com/yu4u/noise2noise/blob/master/model.py#L60-L61

rivergold commented 6 years ago

I meet the same situation as @kawori. When I use u-net, the loss is hard to converge, It decresed fast at first and then decreased slowly and increased to INF suddenly.

yu4u commented 6 years ago

@rivergold hmm... Was it happened on your implementation? On noisy targets' training and/or clean targets' training?

rivergold commented 6 years ago

@yu4u Thanks for your reply. When I use your keras code training on noisy target to remove text in image, the final loss is about 20. And I rewrite the code with TensorFlow train with same parameter, the loss decresed fast at first and then decreased slowly and increased to INF suddenly.

yu4u commented 6 years ago

Thank you for your information. I'm afraid that I could not give useful comment on why TensorFlow implementation does not work while Keras implementation does...

Shakarim94 commented 5 years ago

Did you guys fix the issue of the INF loss? I tried to make the training as close to the original paper as possible. Using the Imagenet validation set and the same U-Net as in paper. But still getting INF loss in the 2nd epoch

@yu4u Did you do any pre-processing such as image normalization, scaling or mean subtraction to avoid this? I personally normalized the images to [0,1], will try to train without any normalization.

Update: tried training without normalizing to [0,1], still getting INF loss Update2: I added Batch Norm layers to the UNET and that seems to have solved the INF loss issue. Now I only need to understand why your network performs well even without Batch Norm :)

yu4u commented 5 years ago

Hi, @Shakarim94

Did you do any pre-processing such as image normalization, scaling or mean subtraction to avoid this?

No, the input input and target images are np.uint8 array in [0, 255].