Custom COLMAP datasets

[maturk]

Hi @jkulhanek,

Great work! Was wondering how to run the method on generic colmap based datasets, with e.g. sparse/0 bin files. Any documentation to follow, or is it not supported? Let me know... Thanks!

[jkulhanek]

Hi, @maturk,

It is supported. Does it not work when you point --data <folder> to the folder with your dataset (images and sparse folder)? You can follow the documentation here: https://jkulhanek.com/nerfbaselines/docs/using-custom-data.html

[maturk]

Oppsie, user error, thank you :) Just as a general question as well, I notice you do not support having just the 2D appearance model enabled. For the ablation Table 4 comparing w/o Gaussian embeddings, is the network architecture for output color then just (beta, gamma) = f(per_image_embedding, C_base_color) and C_out = gamma * C_full_color + beta?

I am mainly wondering since having just the appearance_model_2D enabled (no per-gaussian features) would make sense if the only parameter changing during image capture is e.g. the exposure settings, since there would be no other environmental or dynamic changes.

[jkulhanek]

I see. We didn't make the appearance_model_2D code part of the public release to simplify the code for users. But I can add it if people would find it useful.

[maturk]

I would at least find it useful, but I think I understood the idea to implement it myself too, so no worries! :) I am anyways trying to port some of your appearance stuff to gsplat for testing :)

[maturk]

@jkulhanek Btw i think there is a discrepancy in either the code or the paper.

The paper states: We input the per-image embedding e_j , per-Gaussian embedding g_i, and the base color c_i (0-th order SH) into an MLP which indicates that c_i is of shape (n_gaussians, 3)

But the code in L:1592 takes in the full shape of features which is (n, 48) which is the size of features_rest which is the full SH coefficients, not just the base color? Is my understanding correct?

[jkulhanek]

There is a flag which controls if only the base color is used. Here: https://github.com/jkulhanek/wild-gaussians/blob/481e46b478fc259e991bf4873b139c0ad33613ec/wildgaussians/method.py#L894 it selects only sh0 if enabled. By default only base color (sh0) is used - as in the paper. Or am I missing something?

[maturk]

Got it, @jkulhanek. Makes sense! I managed to implement your proposed EmbeddingModel with gsplat, it works well.

I found it interesting what happens to the raw_colors if you don't use both raw_colors and color_corrected_colors in the loss function (eq. (6) raw_colors with dssim and cc_colors with L1).

Here is a render of the raw_colors when you only use cc_colors in the losses:

And here is the same raw_colors when you separate the dssim and L1 losses as you propose in eq. (6):

gt image for reference:

I'll also test without gaussian features, since my data should not have any environmental/dynamic changes...

[jkulhanek]

Thanks a lot for sharing! If there is not much app. changes in the data, using just cc in the losses would mean the original raw colors can drift away - which you can see in your images. In my experiments, using the raw colors for ssim helps to "fix" the drift. But more importantly, I noticed using just cc colors messes a bit with the grad statistics and makes the growth/pruning less stable.