Question on the post_tune argument on the novel view synthesis

[nsarafianos]

Thank you so much for putting this code up and for providing clear instructions on how to use it. I'm testing the novel view synthesis code on some real images and I have got it to work but without the --post_tune parameter. The moment I add it to the argument I cannot get it to work since the dataloader on the part of the code is empty (error is that batchSize is 0 there).

Hence I wanted to ask you:

1. Is post_tune important as a parameter in terms of the quality of the final output?
2. If it is important could you please help me getting it to work. Where are the outputs "saved" as pair before the post_tune and how could we assign them to the corresponding paths that are read from.

Again thanks again I greatly appreciate it Nikolaos

[StevenLiuWen]

Hi, @nsarafianos, the motivation for us to use --post_tune is going to improve the generalization of our network, since we train our model on the iPER dataset, and the diversity of this dataset is not enough to handle the diverse inputs. Thus, here, we use a trick --post_tune, and it just finetunes the network with a few steps on a test input image.

1. However, the --post_tune trick does not always get work and result in a more decent result;

2. Would you mind provide the snapshot of the error messages for us? Maybe we can help you to run it successfully.

[nsarafianos]

Hi @StevenLiuWen thank you for getting back to me. The command I run is this one.

python run_view.py --gpu_ids 0 --model viewer --output_dir ./outputs/results/ --src_path IMG_PATH --bg_ks 13 --ft_ks 3 --has_detector --front_warp --batch_size 1 --post_tune

When including the --post_tune argument these 2 guys len(self.im_pair_list) == len(self.all_pkl_paths) in the MetaCycleDataSet class are empty as the error below indicates.

Traceback (most recent call last): File "run_view.py", line 49, in <module> adaptive_personalize(opt, viewer, visualizer) File "/impersonator/run_imitator.py", line 216, in adaptive_personalize loader = make_dataset(opt) File "/impersonator/run_imitator.py", line 200, in make_dataset drop_last=True) File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 219, in __init__ batch_sampler = BatchSampler(sampler, batch_size, drop_last) File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 190, in __init__ "but got batch_size={}".format(batch_size)) ValueError: batch_size should be a positive integer value, but got batch_size=0

My assumption is that you guys are assuming to be first writing out something (to the pairs and imgs subfolders under results) that this dataloader is then supposed to read from but potentially nothing is written out until later on the run_view.py file and hence these folders are empty (?) However I have not got it to work and I'm not sure what needs to go where exactly so if it's easier for you, you can just tell me what needs to go where and I can write them out.

Again thank you so much for looking into it.

[StevenLiuWen]

Hi, @nsarafianos; yes, I think you might ignore some resources which need to be downloaded first before you could run the --post_tune, https://github.com/svip-lab/impersonator#download-resources (step 3).

You need to ensure that you have to download the samples.zip file, unzip them, and move them to the assets folder.

wget -O assets/samples.zip "https://1drv.ws/u/s\!AjjUqiJZsj8whLNz4BqnSgqrVwAXoQ?e=bC86db"

mv samples  assets

unzip  assets/samples.zip

Finally, you will find the folder ./assets/samples/A_priors/imgs.

By the way, the --post_tune trick mainly dose the following things:

1. for each input test image, we first run its inference (pre-trained network) with respect to a pivot person (./assets/samples/A_priors/imgs) who turn around and get the synthesized results as the pseudo-ground-truth images;

2. then, we finetune the pre-trained network several steps with the pseudo-ground-truth images. Since the input test image provides some visible part of textures, we both use the input test image and the pseudo-ground-truth images as the reconstruction loss functions to improve the generalization of the network.

This trick does not always get work, and how to improve the generalization of the generative models is a problem to be further studied in the research community, especially in the field of neural rendering.

[nsarafianos]

Oops I'm sorry I skipped that part. It now works thank you so much for your suggestions. As for the quality and the generalization yes definitely it's a hard problem far from solved but currently even if the quality is not perfect it's good enough for something I'd like to try.

Again thank you and congrats for this nice paper.