search

VITA-Group / EnlightenGAN

[IEEE TIP] "EnlightenGAN: Deep Light Enhancement without Paired Supervision" by Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, Zhangyang Wang
Other
918 stars 200 forks source link

Randomness of training #87

Closed Carlyx closed 3 years ago

Carlyx commented 3 years ago

Hi, your work is amazing！

I encountered some problems during training. The same code is trained twice on the same NVIDIA GPUs, but the result is always not fully reproduced. For example, the average NIQE may differ by 0.05. So how to determine whether the performance increase or decrease is due to the structure or the randomness of the result when comparing?

(I have tried to fix the random seed and using a more stable upsampling method. )

Could you give me some suggestions? Looking forward to your reply.

yifanjiang19 commented 3 years ago

I think 0.05 difference in NIQE is acceptable

Carlyx commented 3 years ago

I see, thank you for your reply~

McFly-Jia commented 3 years ago

Hi, your work is amazing！

I encountered some problems during training. The same code is trained twice on the same NVIDIA GPUs, but the result is always not fully reproduced. For example, the average NIQE may differ by 0.05. So how to determine whether the performance increase or decrease is due to the structure or the randomness of the result when comparing?

(I have tried to fix the random seed and using a more stable upsampling method. )

Could you give me some suggestions? Looking forward to your reply.

Hi, your work is amazing！

I encountered some problems during training. The same code is trained twice on the same NVIDIA GPUs, but the result is always not fully reproduced. For example, the average NIQE may differ by 0.05. So how to determine whether the performance increase or decrease is due to the structure or the randomness of the result when comparing?

(I have tried to fix the random seed and using a more stable upsampling method. )

Could you give me some suggestions? Looking forward to your reply.

我也测试了作者的结果，但是没能得到跟作者论文中一样的结果，而且相差很大，您可以发我一下您所用的测试代码吗？

Carlyx commented 3 years ago

Hi, your work is amazing！ I encountered some problems during training. The same code is trained twice on the same NVIDIA GPUs, but the result is always not fully reproduced. For example, the average NIQE may differ by 0.05. So how to determine whether the performance increase or decrease is due to the structure or the randomness of the result when comparing? (I have tried to fix the random seed and using a more stable upsampling method. ) Could you give me some suggestions? Looking forward to your reply.

Hi, your work is amazing！ I encountered some problems during training. The same code is trained twice on the same NVIDIA GPUs, but the result is always not fully reproduced. For example, the average NIQE may differ by 0.05. So how to determine whether the performance increase or decrease is due to the structure or the randomness of the result when comparing? (I have tried to fix the random seed and using a more stable upsampling method. ) Could you give me some suggestions? Looking forward to your reply.

我也测试了作者的结果，但是没能得到跟作者论文中一样的结果，而且相差很大，您可以发我一下您所用的测试代码吗？

我用了作者开源的代码，没有进行任何结构或参数上的改动。 不同的环境，不同的服务器，不同的显卡以及GAN本身都会引入一些不确定的因素。我采用了预训练模型 以及 从头训练两种方式，最终的结果都不尽相同但接近。并且在测试时，图片的分辨率也会影响结果。

具体数据上， 在使用预训练模型在DICM（64）上的测试NIQE得到3.5522的结果，和paper中相近。 从头训练一遍后在DICM（64）上测试NIQE得到3.682的结果。

在使用预训练模型在所有测试集（跟paper中提到的不完全一致）上的平均NIQE为3.6281 从头训练一遍后在所有测试集（跟paper中提到的不完全一致）上的平均NIQE为3.6155

yifanjiang19 commented 3 years ago

To reproduce the results from scratch, I would suggest to use CUDA 9.0 and pytorch 0.3.1. For some unkown issue, cuda>10 will cause different results as our pretrained checkpoint.