Number of Rays used for importance Sampling

[wcy199705]

Thanks for your good works! I have a question about the number of rays sampled in importance sampling (IST/ISG). Dynamic regions are well reproduced, as the paper claims. However, in my code, the outdoor area (window) does not reproduce well, contrary to the results of the paper. I use 1K resolution images( 1014 x 1352 ). I sample 24576 rays per image (importance sampling) and use 8192 rays per iteration (3 iterations(backpropagation) per image). Could you tell me how many rays are used for importance sampling (ISG/IST)?

Thanks.

[violetamenendez]

Hi @wcy199705 , have you released your code anywhere? :-)

[wcy199705]

Hi @violetamenendez, Because my codes do not reproduce the paper, I have not released my implementations of this paper. When I finished my current project, I want to make my codes public. To perfectly reproduce the results of the paper, I would like to know implementation details such as the number of rays sampled in importance sampling.

[zhaoyang-lv]

Not sure I get your questions. Do you mean how many rays we sample per batch? Does this closed issue answer your question? https://github.com/facebookresearch/Neural_3D_Video/issues/10

For the ray importance sampling per batch, you calculate the weights for all the rays per image, and sample the number of rays per batch according to the weights using importance sampling.

[wcy199705]

Thanks for your reply! @zhaoyang-lv I confuse in the number of rays to perfectly reproduce. In the implementation details of the paper, the total batch size is 24576 rays. Is the total batch size equal to the number of rays sampled in importance sampling per image? However, in your answer #10, the total batch size seems to be 8 x 1024 rays (8 NVIDIA V100 GPUs which is used in the paper). Which is right ( 8 x 1024 rays or 24576 rays)? Thanks for your kind helps!

[zhaoyang-lv]

It depends on the GPU being used. We did a lot of experiments and mixed V100 GPU (with 32GB memory each) with P100 GPU (with 16 GB memory each). For the paper we refer to the V100 GPU setting. For all the training parameters in the paper, you can refer to that. Honestly I don't find much difference in either setting for this batch size. It might affect convergence, only if you use too small batch (<1024 x 4 rays).

[wcy199705]

Thanks for your fast reply!! @zhaoyang-lv The number of rays sampled in importance sampling per image is 24576 in the paper? I think the number of rays sampled in importance sampling affect the result. Because, in ISG keyframe importance sampling, the outdoor(windows in the scene) seems to be not sufficiently sampled. If the number of rays sampled in importance sampling is large, then low weight regions (e.g. windows) would be sampled many times. The picture is weight of ISG sampling. Thanks for the great help!

[zhaoyang-lv]

The number of rays sampled in importance sampling per image is 24576 in the paper?

Yes.

I think the number of rays sampled in importance sampling affect the result.

It will surely affect. But it should also offset the effect if you just do longer iterations with smaller batches. In fact, the total training time might be the very similar if you use smaller batches and longer iterations. Your weight map definitely look reasonable. Since we starts training from key-frames, the background is usually well initialized from the key-frames. It is reasonable to use relatively sparse sampling in the background region.

[wcy199705]

Thanks for your replies and help! @zhaoyang-lv I will try again according to your comments!!