Whether to conduct data cheating in the experiment

[lzg1988]

1. sid and did use 0/ for training and you use 0/ +2/* for training?
2. sid and did use data config(0.1+0.04->10/30) and you use (0.1->10/30)? more training data and easier data complexity, is that fair????

You reply https://github.com/MohitLamba94/LLPackNet/issues/4#issuecomment-1345939280

1. testing issue

# from https://github.com/MohitLamba94/LLPackNet/blob/master/ablations/LLPackNet.ipynb
class get_data(Dataset):
    """Loads the Data."""

    def __init__(self,opt):
        self.train_files = glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/short/1*_00_0.1s.ARW')
#         self.train_files = self.train_files + glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/short/2*_00_0.1s.ARW')

        self.gt_files = []
        for x in self.train_files:
            self.gt_files =self.gt_files+ glob.glob('/media/mohit/data/mohit/chen_dark_cvpr_18_dataset/Sony/long/*'+x[-17:-12]+'*.ARW')

        self.to_tensor = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])

        self.opt = opt

you used only 0.1s raw data for testing, why not use the below data for testing???

10003_00_0.04s.ARW  10030_00_0.04s.ARW  10045_00_0.04s.ARW  10178_00_0.04s.ARW  10193_00_0.04s.ARW  10217_00_0.04s.ARW
10006_00_0.04s.ARW  10032_00_0.04s.ARW  10054_00_0.04s.ARW  10185_00_0.04s.ARW  10198_00_0.04s.ARW  10226_00_0.04s.ARW
10011_00_0.04s.ARW  10034_00_0.04s.ARW  10055_00_0.04s.ARW  10187_00_0.04s.ARW  10199_00_0.04s.ARW  10227_00_0.04s.ARW
10016_00_0.04s.ARW  10035_00_0.04s.ARW  10068_00_0.04s.ARW  10191_00_0.04s.ARW  10203_00_0.04s.ARW  10228_00_0.04s.ARW
10022_00_0.04s.ARW  10040_00_0.04s.ARW  10069_00_0.04s.ARW  10192_00_0.04s.ARW  10213_00_0.04s.ARW
10178_00_0.033s.ARW  10187_00_0.033s.ARW  10192_00_0.033s.ARW  10198_00_0.033s.ARW  10203_00_0.033s.ARW  10217_00_0.033s.ARW  10227_00_0.033s.ARW
10185_00_0.033s.ARW  10191_00_0.033s.ARW  10193_00_0.033s.ARW  10199_00_0.033s.ARW  10213_00_0.033s.ARW  10226_00_0.033s.ARW  10228_00_0.033s.ARW

which was tested in sid and did.

# from https://github.com/parasmaharjan/DeepImageDenoising/blob/master/Pytorch_EDSR_Test.py#L120
in_files = glob.glob(input_dir + '%05d_00*.ARW' % test_id)
    for k in range(len(in_files)):

2. training issue

you said

For training I merge the 0 and 2 both of which belong to training phase

But for sid and did, they only use 0* data, totaly 161. BUT you use 181 data(more 20 data) for training?

# from https://github.com/cchen156/Learning-to-See-in-the-Dark/blob/master/train_Sony.py#L17
train_fns = glob.glob(gt_dir + '0*.ARW')
train_ids = [int(os.path.basename(train_fn)[0:5]) for train_fn in train_fns]

# from https://github.com/parasmaharjan/DeepImageDenoising/blob/master/Pytorch_EDSR.py#L39
train_fns = glob.glob(gt_dir + '0*.ARW')
train_ids = []
for i in range(len(train_fns)):
    _, train_fn = os.path.split(train_fns[i])
    train_ids.append(int(train_fn[0:5]))

Can you explain the above two issues?:)

[lzg1988]

https://github.com/MohitLamba94/LLPackNet/issues/4#issuecomment-1347631187

Anyway, here are further discussions to figure out whether you work(LLPackNet BMCV 2020 and RED CVPR2021) are replicable.

Refer to Understanding Health Research, "It is very important that research can be replicated, because it means that other researchers can test the findings of the research. Replicability keeps researchers honest and can give readers confidence in research."

So

1. Can your provided model weights(LLPackNet and RED) reproduce your good results in paper
2. As you mentioned, you use 0 and 2 for training. So, follow your advice, if we use all(including 0.1s+0.033s + 0.04s) 0 and 2 data for training and your provided training code in LLPack github and RED, can we repruduce your work in papers LLPackNet BMCV 2020 and RED CVPR2021?

[lzg1988]

https://github.com/MohitLamba94/LLPackNet/issues/4

[lzg1988]

Thanks for your patience!! :)

Maybe there is no much difference between using 0+2 and 0* for training.

However, the main issue is whether you use ONLY 0.1s data for training and testing. According to our experiments, we retrain sid using all data(0.1s+0.033s+0.04s), the psnr and ssim(28.89/0.787) is almost the same as their paper. But if we use 0.1s data ONLY, the psnr and ssim is much better(29. 5/0.814).

SO can your pretrained model reproduce your psnr and ssim using ALL testing data(0.1s+0.033s+0.04s) and is your model in your paper training and testing ALL data(0.1s+0.033s+0.04s) as SID or DID?

[lzg1988]

And we also notice other researcher also point out the same issue in RED. So we are also want to figure out whether your result is training and testing in all data.

https://github.com/MohitLamba94/Restoring-Extremely-Dark-Images-In-Real-Time/issues/6

[lzg1988]

There is some strange thing in you work RED https://github.com/MohitLamba94/Restoring-Extremely-Dark-Images-In-Real-Time/commit/31248de859d15489103c40b2a32b480f6fa30f39 In 2022 May 16, you delete your detailed training path and testing paths(USE ONLY 0.1s data) and replace with some confusable '/path_to_SID_short_training'. We don't know what data you TRUELY train and test?

[MohitLamba94]

Hello, so I have done some quick experiments and retrained the model now WITHOUT using the validation set to really see if the validation set is bringing any significant advantages. The exact details are as below,

1. Trained the model only on "000.ARW".Please note now I am using less training data than otherwise. Had to do because now operating with limited RAM resources.
2. But test on full "1*.ARW" test data. This includes all 0.1s, 0.04s and 0.033s exposures.
3. To save time trained only for 500k iters rather than 1000k iters
4. I further do not use better augmentations that have come up for Bayer filters.

And I am yet able to reproduce the same results i.e 0.78 million parameters with 59 GFlops with PSNR = 28.64 dB and SSIM = 0.79

I have further examined the images qualitatively and they look as good.

I hope this helps you and removes any ambiguity.

[MohitLamba94]

There is some strange thing in you work RED 31248de In 2022 May 16, you delete your detailed training path and testing paths(USE ONLY 0.1s data) and replace with some confusable '/path_to_SID_short_training'. We don't know what data you TRUELY train and test?

I hope my previous comment clarifies. The edits were made to tell which part of the code does what.

[lzg1988]

Thanks for your reply. Could you provide your trained model? Thanks~