Finetune in scan118 error

[jerryxu9905]

Hi, I have succeeded to extract correct mesh in Generic mode, but in Finetune mode, I found the extracted mesh become confusing during training.

At the begining of Finetune (0 iters):

End of Finetune (11k iters):

It seems that the Finetune in scan118 degenerate, I don't modify any config or source code, so what's problem? Can anyone who encounter same issue share some solution? Many Thanks!

[flamehaze1115]

Hello. Does the "weight_sum" become 0 in fine-tuning? If so, you can adjust the two parameters "visibility_beta, and visibility_gama" used in fine-tuning.

[youmi-zym]

Hi @flamehaze1115 @xxlong0 @jerryxu9905 , I find another bug that happens at dtu_fit.py:

if self.split == 'train':
            render_idx = self.train_img_idx[idx % self.n_views]
            support_idxs = [render_idx]

The query/reference and src views are the same, which means whatever the sdf_network.color_blend predicts, the network will always output the ground-truth color. As a result, there is no supervision for SDF and depth. For example, for points along the same ray in the reference view, as the intrinsic and extrinsic parameters of reference and src views are the same, the points along the ray will be projected to the same pixel in the reference image, whatever the depth is. Therefore, each ray after pixel warping takes the ground truth color and the photometric loss will be 0.

The solution I tried is:

if self.split == 'train':
            render_idx = self.train_img_idx[idx % self.n_views]
            support_idxs = [i for i in self.train_img_idx if i != render_idx]

Even though it didn't solve the problem during fine-tuning, it works in my refactored code (based on your framework and pixel/patch-based loss).

Hope my finding could help, correct me if I'm wrong! Thanks.

[xxlong0]

You are right youmi. This is the bug that cause finetuing fails. Sorry for the inconvenience. I push the codes rushly, i should have checked the codes carefully to avoid the inconvenience.

Best Xiaoxiao

---

发件人: youmi @.> 发送时间: Tuesday, September 13, 2022 11:46:13 PM 收件人: xxlong0/SparseNeuS @.> 抄送: Long Xiaoxiao @.>; Mention @.> 主题: Re: [xxlong0/SparseNeuS] Finetune in scan118 error (Issue #17)

Hi @flamehaze1115https://github.com/flamehaze1115 @xxlong0https://github.com/xxlong0 @jerryxu9905https://github.com/jerryxu9905 , I find another bug that happens at dtu_fit.pyhttps://github.com/xxlong0/SparseNeuS/blob/main/data/dtu_fit.py#L223:

if self.split == 'train': render_idx = self.train_img_idx[idx % self.n_views] support_idxs = [render_idx]

The query/reference and src views are the same, which means whatever the sdf_network.color_blendhttps://github.com/xxlong0/SparseNeuS/blob/main/models/sparse_neus_renderer.py#L283 predicts, the network will always output the ground-truth color. As a result, there is no supervision for SDF and depth. For example, for points along the same ray in the reference view, as the intrinsic and extrinsic parameters of reference and src views are the same, the points along the ray will be projected to the same pixel in the reference image, whatever the depth is. Therefore, each ray after pixel warping takes the ground truth color and the photometric loss will be 0.

The solution I tried is:

if self.split == 'train': render_idx = self.train_img_idx[idx % self.n_views] support_idxs = [i for i in self.train_img_idx if i != render_idx]

Even though it didn't solve the problem during fine-tuning, it works in my refactored code (based on your framework and pixel/patch-based loss).

Hope my finding could help, correct me if I'm wrong! Thanks.

― Reply to this email directly, view it on GitHubhttps://github.com/xxlong0/SparseNeuS/issues/17#issuecomment-1245601848, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AORGW2O7J42EET43KL64YFLV6COULANCNFSM6AAAAAAQESLQHY. You are receiving this because you were mentioned.Message ID: @.***>

[jerryxu9905]

Hello. Does the "weight_sum" become 0 in fine-tuning? If so, you can adjust the two parameters "visibility_beta, and visibility_gama" used in fine-tuning.

Hello @flamehaze1115. Yes, the weight_sum become 0 in fine-tuing,I have tried many times for visibility beta and visibility gama params, but still can't get a reliable result, let alone every 15 scenes try shall cost so much time.... Can you share your visibility beta and visibility gama settings in every 15 test scenes? It will help us save lot of time.

Many thanks.

[flamehaze1115]

Hello. Does the "weight_sum" become 0 in fine-tuning? If so, you can adjust the two parameters "visibility_beta, and visibility_gama" used in fine-tuning.

Hello @flamehaze1115. Yes, the weight_sum become 0 in fine-tuing,I have tried many times for visibility beta and visibility gama params, but still can't get a reliable result, let alone every 15 scenes try shall cost so much time.... Can you share your visibility beta and visibility gama settings in every 15 test scenes? It will help us save lot of time.

Many thanks.

After fixing the dataloader bug, still have the problem?

[jerryxu9905]

After fixing the dataloader bug, still have the problem?

@flamehaze1115 Yes, I'm using the latest code where in dtu_ft.py

if self.split == 'train':
            render_idx = self.train_img_idx[idx % self.n_views]
            support_idxs = [i for i in self.train_img_idx if i != render_idx]

shell command is:

#!/usr/bin/env bash
python exp_runner_finetune.py \
--mode train --conf ./confs/finetune.conf --is_finetune \
--checkpoint_path ./weights/ckpt.pth \
--case_name scan118  --train_imgs_idx 0 1 2 --test_imgs_idx 0 1 2 --near 700 --far 1100 \
--visibility_beta 0.015 --visibility_gama 0.010 --visibility_weight_thred 0.7

scan118 finetune: 0 iters -> 1000iters -> 2000iters

[flamehaze1115]

Ok. I will check the code and back asap.

[zjhthu]

I find removing the Consistency loss term can resolve this problem, although this might give suboptimal results as shown in the paper. More specifically loss = torch.mean(weight[mask] * error) in this line and term2, term3 = 0, 0.

[flamehaze1115]

Hello guys. Sorry for the late. I am very busy this week. I have updated the ft bash. The old parameters should be not proper. The consistency-aware loss is sensitive, so we introduce another parameter "visibility_weight_thred" to dynamically adjust the beta param. This param will increase the beta in the training if the "weight_sum" becomes too small.

#!/usr/bin/env bash python exp_runner_finetune.py \ --mode train --conf ./confs/finetune.conf --is_finetune \ --checkpoint_path ./weights/ckpt.pth \ --case_name scan118 --train_imgs_idx 0 1 2 --test_imgs_idx 0 1 2 --near 700 --far 1100 \ --visibility_beta 0.025 --visibility_gama 0.010 --visibility_weight_thred 0.7 0.6 0.5

[FENGGENYU]

Any update on this issue? I still get empty mesh after fine-tuning, even it is from the latest bash.

The shape mesh degenerates after fine-tuning.

[guangxuwang]

still same error