Edge fitting

[ZhouCX117]

Thanks for your excellent! How could I use the EMAP code to fit curves and lines from the trained 3DGS? Now I regard the Gaussian centers and directions as points and linedirection and feed them into the get_parametric_edge function of EMAP https://github.com/cvg/EMAP/blob/main/src/edge_extraction/extract_parametric_edge.py. Is it right?

[kunalchelani]

Hello! Yes, that would be one way to do it. However, EMAP's method of clustering of points with direction, and also ours is fitted to the distribution of points and associated direction (and noise in it) obtained from the underlying process (voxel-based UDF and Gaussians respectively). So, if you haven't applied the geometric regularization to the Gaussians mentioned in our paper, you would get a slightly different distribution of Gaussian centers. It is however worth trying out the steps you mention to establish one baseline. Curious to know how it goes!

[ZhouCX117]

I have applied the geometric regularization to the initial Gaussians and utilized the EMAP's code to fit curves and lines. The results are bad. It seems the trained Gaussians are noisy. Could you please provide the code described in Section 3.3?

[kunalchelani]

We will be releasing the code soon. Thanks for the patience!

[Jiahao-Ma]

Thanks for your awesome work. I am still curious about edge filtering. Some Gaussians do not lie on the edge. Should we first filter and select only those Gaussians on the edge before applying EMAP's method to extract the edge, or should we apply EMAP's method to all Gaussians?

[kunalchelani]

Filtering is indeed helpful if there are wayward Gaussians. You can also fit edges to all Gaussians and then filter out edges that do not project to edges in the 2D edge maps, how it was done in EMAP. In our work, we performed opacity based filtering, along with filtering statistical outliers. For ABC-NEF objects, we also filtered Gaussians not projecting to an edge in at least a minimum percentage (5-10%) of total views.

[kunalchelani]

Added code. Please feel free to re-open if there are more issues, or you'd want to discuss.

[ZhouCX117]

@kunalchelani Thanks for your sharing. I am trying to reproduce the results. However, in line 36 of 'fit_edge.py', the function 'init_views_and_get_scale' is not available in the train_utils.py file. Could you please check it? Thanks again!

[kunalchelani]

Thanks for pointing out. I had missed a refactoring change. Let me know if there are still issues with the latest commit.

[ZhouCX117]

@kunalchelani Thanks for your reply. It works now! Excellent results! By the way, could you please give a more detailed guidance for evaluation? Step1: Edge extraction and Step 2: eval.py? How about the config setup such as 'use_parametric_edges' and 'use_filtered_points'? And is the process same for ABC and DTU datasets?

[ZhouCX117]

Sorry to bother you again. Could you please provide the JSON file for both PidiNet and DexiNed edge detectors?

[kunalchelani]

@ZhouCX117 Regarding the evaluation, you should use the flag --use_parametric_edges as it uses the points sampled at the desired resolution on the parametric edges. The flag --use_filtered_points is for evaluating the raw Gaussian means, but they can be denser or sparser than the ground truth sampling and therefore this evaluation is not completely correct to do. I will remove this option in the next commit. The eval.py in its current state is meant for evaluation over ABC. As mentioned in the README, evaluation over DTU can done using the referred EMAP script.

As far as the configs are concerned, I will upload the PidiNet config for ABC after testing that it produces roughly the same results as mentioned in the paper, but I can do this some time next week.

[ZhouCX117]

@kunalchelani Thanks for your reply. Looking forward to seeing the update. By the way, could you please provide the render file?

[kunalchelani]

@ZhouCX117, I'm sorry I didn't get which render file you're talking about.

[ZhouCX117]

@kunalchelani Thank you for your response! I need a file similar to https://github.com/graphdeco-inria/gaussian-splatting/blob/main/render.py for visualization.

[kunalchelani]

I see, I currently don't save the rendered images from the views, I only use the tensorboard logger to visualize these images. I might add the functionality later. I guess you can use something similar to get_output function after loading the viewpoints and the learnt Gaussian params, to render the images.

[Chauncywu1999]

@kunalchelani Thanks for your great work! When I trained using the DTU dataset with scan118, the training process ran without issues. However, an error occurred during the edge fitting process, as shown in the image below. I don't know how to figure out this.

[kunalchelani]

@Chauncywu1999 The warning is most likely not the cause of not getting curves. Are you sure you are using the correct config file for DTU (config/DTU.json) ? The parameters line_curve_residual_comp_factor and angle_thresh in the parametric_fitting section of the config affect the inclination towards more lines or edges. Upon testing at my end, I get reasonable number of curves with the default configuration in DTU.json.

[Chauncywu1999]

@kunalchelani Yes, I did not change the DTU.json. This is the result of edge extraction that I have achieved so far, which feels like there are fewer lines compared to the original paper.

[kunalchelani]

This looks quite similar to the output in the paper to me. Perhaps, a few edges less which can be because of the duplication and culling, which at the moment is at fixed steps. This is one of the shortcomings that the current pipeline is prone to hyper-parameter tuning but that is the case for the original Gaussian Splatting paper as well, on which this is based. Anyway, you can try with this config, gives a few more edges (I haven't checked the reproducibility for DTU yet, will do that soon). DTU.json

[ZhouCX117]

@kunalchelani Hi, would you please provide the trained checkpoints and the JSON files for the TT and Relipca datasets?