Understanding the fetching of mlp weights for the color network

[Shubhendu-Jena]

Hi,

Thank you for the wonderful work. I was trying to understand the procedure of obtaining mlp weights. While it was straightforward for the density network (i.e. grid sampling), I was having a hard time understanding the same for color network. I see that instead of obtaining features by using interpolation directly from the planes, you instead obtain the weights that should be used to interpolate the final colors (please correct me if I am wrong). For this, you obtain query_indices and query_weight from the function self.get_batch_size_per_network. I'd be grateful if you could briefly describe the logic used in obtaining the color network weights from the planes and how you obtain the final color per pixel.

Thanks in advance

[yunzhiy]

Hi, thanks for you interest. For the final pixel color, we follow KiloNeRF to use the HPC MAGMA library to query tiny color mlps simultaneously. So we need to reorder feature vectors of input 3D points according to the tiny mlp indices they belong to and aggregate the results via summation with weights. In our actual implementation, we map each 3D point to four tiny mlps on one mlp map and bilinearly interpolate the color (Your understanding of query weight is correct).

You can also read this part of the KiloNeRF code if you have any other questions.