Difficulties on reproducing

[pjessesco]

Hi, thanks for sharing this amazing work. I'm trying to reproduce DeProCams, and having some trouble to get plausible result. I'd like to ask some hints to solve...

Here is my detailed workflow to reproduce DeProCams, basically following described instruction in readme.

1. Calibrate using your other work.

   • I captured 11 image pairs, and calibration looks quite well comparing our configuration with visualized extrinsics.
   • Camera, projector and stereo reprojection error is 0.819, 1.787, 1.390 respectively.
   • I did not check distortion option since camo_cmp example seems not using too.
   • Image resolutions are 800x600 for projector and 640x480 for camera.

2. Write param.yml manually using displayed values in Matlab main window.

3. Run DeProCams with captured dataset

And here is our estimated relighting image,


and estimated relighting image in earlier iteration.


I was confused that my results are quite different with your several results. I tried in several different setups and similar results are reproduced.

My guess is that mismatch of an initial shape of projection from network (red in below image, vertically long rectangle) with an actual projection shape (yellow rectangle in below image) affects to result quite a lot, but I have no idea which factors affects this results...

mismatch between 2 shapes


I also tried to train with overfitting (i.e. num_train_list=[1]), and it seems works.

relighting image (overfitting)


Any suggestions would be appreciated, thanks for reading.

[BingyaoHuang]

My guess is that mismatch of an initial shape of projection from network

Could be, can I see the mask image?

[pjessesco]

Thanks for reply, here is mask image from loadData() function, I increased thresh to 50.

Results was similar comparing with thresh=10 (images in my first comment)

Relighting image (thresh=50)


[BingyaoHuang]

The reflection from the desk is misdetected as direct light, maybe this is why the mismatch between 2 shapes. A simple verification is to cover the reflective region with a rough rug, then recapture the data and check whether the direct light mask is correct. Finally, try retraining DeProCams.

If its due to the wrong direct light mask, a more robust direct light mask detection would be using shifted checkerboard patterns. You can also check the exemplar checkerboard images in setups\camo_cmp\cam\raw\cb.

[pjessesco]

Thanks for your kindness support and suggestion 👍