search

DY112 / LSMI-dataset

[ICCV'21] Large Scale Multi-Illuminant (LSMI) Dataset for Developing White Balance Algorithm under Mixed Illumination
https://www.dykim.me/projects/lsmi
58 stars 3 forks source link

train val test split for single illuminant and mixed illuminant training schedules #15

Closed CemEntok closed 1 year ago

CemEntok commented 1 year ago

Hello

is there any JSON file including train/val/test splits you can share publicly so that your single illuminant and mixed illuminant results can be reproduced? I noticed that the files you shared in separate_json_files are only consisting of multi-illuminant images.

DY112 commented 1 year ago

Hello. There is a slight discrepancy between the number of instances we used to generate the numbers reported in the paper at the time of publication and the results our current code produces.

The single illuminant results reported in the paper used sub-image (_1) of the 99 two-illuminant test scene, and the multi-illuminant used the two/three-illuminant test scene (_12,_123). The code has now been updated to handle all the sub-images (_1,_12,_13) of the three-illuminant scene as well.

If you want to generate slightly reduced test instances to reproduce our result, please refer to following code: https://github.com/DY112/LSMI-dataset/blob/4e12d52ad6a17a0b7350113fb1f287263ac096ec/2_preprocess_data.py#L47C1-L48C25

This is how we initially processed the test scene, and it generates a test set that matches what is reported in the paper.

CemEntok commented 1 year ago

Thank you for your reply!

  1. So your results are obtained from the test set consisting of:

So by looking at these bullet-ins, you test the model with data A for obtaining single illuminant results, data B+C for obtaining multi-illuminant results, and data A+B+C for obtaining mixed-illuminant results so that the numbers shared in the paper are matched:

image

Am I approaching the testing of the model in your paper in the right manner? Correct me please if I am wrong.

  1. Did you train the model with two_iilum_train and three_illum_train places and then test the model with the test mentioned above or did you change the training set for every test case such as separating the training set for each case of single illuminant, multi-illuminant and mixed illuminant cases? If so, how did you separate your training set for each case?
  1. The last point is, that you did not include _1, _12, _13 of the 3-illuminant scenes in the results of the paper. Therefore, if I had included them in the training, I could not reproduce the results I suppose.
DY112 commented 1 year ago

Sorry for late reply.

  1. You are right.
  2. Every evaluation results are from single model which is trained with full train set.
  3. You are right. We provide option flag whether to include sub-images of 3-illuminant scenes or not.