Albedo and Shading incorrect in Composer output

[akshay-iyer]

Hello,

While I'm trying to run the composer with the same code, I'm getting a good reconstruction but we can see that the albedo image has shading artifacts and the shading image is mostly white and flat. I trained it for 108 epochs with each epoch having the schedule - 10_shape,10_shading,reflectance,20_lighting

https://drive.google.com/file/d/1CpjuBO7jbZvlkDE3zOcoBVoXfQC5j2ft/view?usp=sharing

I could see overfitting in the albedo, shading and reconstruction loss. Hence I added dropout in the decomposer and trained it for 500 epochs. Then I trained the composer network with dropout added in decomposer and shader and made lights multiplier equal to 1.5. This was with the same schedule as above for 300 epochs The overfitting reduced but the network learned poorly on the albedo but did relatively well on shape and shading from before. The albedo output from the decomposer is good but not from the composer. As we can see below:

https://drive.google.com/file/d/1bYfbIxrFp4zPSS02feMNfdzh0Xtrmxsj/view?usp=sharing

So now I'm training the albedo decoder of the composer independently. Kindly help me with what is going wrong. In the end, I aim to be performing self-supervised training with images of my own.

[jannerm]

Yeah, it looks like a degenerate solution has been found in which the albedo is being used to explain almost all of the image. To help debug, could you post:

1. Your python version
2. Your PyTorch version
3. The commands you ran to produce the first set of images (and which datasets you trained on)

[akshay-iyer]

Python - 3.6.7 torch - 0.1.12.post2

The decomposer used in the composer was trained on cars and airplanes and the shader was trained on motorbike, airplane and bottles Command used: python composer.py --decomposer saved/decomposer/state.t7 --shader saved/shader/model.t7 --save_path saved/composer/ \ --unlabeled suzanne_train,teapot_train,bunny_train \ --labeled car_train,airplane_train \ --val_sets suzanne_val,teapot_val,bunny_val,airplane_val,car_val \ --unlabeled_array shader --labeled_array shader \ --transfer 10-shader_10-normals,reflectance_20-lights --num_epochs 300 --save_model True --data_path ~/intrinsics-network/dataset/output --set_size 5000 \ --num_val 10

Since, dropout and early stopping didn't help in the cause, I repeated the same run as above with the same saved models and transfer schedule but with more number of images this time to reduce overfitting: (20k per category) --unlabeled airplane_train \ --labeled motorbike_train,car_train \ --val_sets motorbike_val,car_val,airplane_val \

The second run is now complete at 98 main epochs with 40 sub epochs within (10-10-20). I stopped at 98 epochs since the output was not showing any major improvement

Interim validation output after 46 epochs: https://drive.google.com/open?id=10Q0i-Yz0gOP1BymzLmK06jT1FvWx-id1 Final validation output after 98 epochs: https://drive.google.com/open?id=1npgWxJD82kxMG1MRFGlSg5YAxJ8zVm77

Following are the links to the error plots from the two runs. The lights error shoots up really high in the first run. : First run error plots: https://drive.google.com/open?id=1AdV2eXt5BnVbKD2dlLrqZXbptVOy0PFR Second run error plots (after 46 epochs): https://drive.google.com/open?id=1psBThHEMF5Xy1e4r-Xwj93y0JkGp0haJ Second run error plots (final) : https://drive.google.com/open?id=1rYF8LeSedb-sRSFOcbnAx3izacAjDM1r

Unable to come to the error values in the paper. What should be the number of epochs to train the composer since the decomposer and shader perform better individually? Individual decomposer output : https://drive.google.com/open?id=1smpHZLoh-jHq1h4_56xHD-Gnnj63UHZ6 Individual shader output: https://drive.google.com/open?id=1pUGu4uQbOOHkaRE7CgM9V_H4-L69SJGC

Kindly let me know if you need any more information.

Cheers, Akshay