run_custom.py gets stuck on custom dataset

[themozel]

When i run the custom script on my own data it gets stuck in a (Pdb)? state. What am I doing wrong here ?

python run_custom.py --mode run_video --video_dir /home/themozel/uni/BundleSDF/supermarket/ --out_folder /home/themozel/uni/BundleSDF/outputs/ --use_segmenter 1 --use_gui 0 --debug_level 4

Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=20, releaselevel='final', serial=0)
Warp 1.4.2 initialized:
   CUDA Toolkit 12.6, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4070 SUPER" (12 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.4.2
Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=20, releaselevel='final', serial=0)
Warp 1.4.2 initialized:
   CUDA Toolkit 12.6, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4070 SUPER" (12 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.4.2
[2024-11-24 11:12:13.548] [warning] [Bundler.cpp:49] Connected to nerf_port 9999
[2024-11-24 11:12:13.549] [warning] [FeatureManager.cpp:2084] Connected to port 5555
default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}}
Warp UserWarning: Python 3.9 or newer is recommended for running Warp, detected sys.version_info(major=3, minor=8, micro=20, releaselevel='final', serial=0)
Warp 1.4.2 initialized:
   CUDA Toolkit 12.6, Driver 12.4
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4070 SUPER" (12 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.4.2
[bundlesdf.py] percentile denoise start
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start color_20241119_092324_286
[bundlesdf.py] process frame color_20241119_092324_286
[2024-11-24 11:12:16.158] [warning] [Bundler.cpp:268] Added frame color_20241119_092324_286 as keyframe, current #keyframe: 1
[bundlesdf.py] processNewFrame done color_20241119_092324_286
[bundlesdf.py] color_20241119_092324_286 prepare data for nerf
[bundlesdf.py] rematch_after_nerf: True
[2024-11-24 11:12:16.172] [warning] [Bundler.cpp:961] Welcome saveNewframeResult
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
[2024-11-24 11:12:16.318] [warning] [Bundler.cpp:1110] saveNewframeResult done
[bundlesdf.py] percentile denoise start
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start color_20241119_092326_327
[bundlesdf.py] process frame color_20241119_092326_327
[bundlesdf.py] frame_pairs: 1
[loftr_wrapper.py] image0: torch.Size([1, 1, 400, 400])
[loftr_wrapper.py] net forward
[loftr_wrapper.py] mconf, 0.21434098482131958 0.5707454681396484
[loftr_wrapper.py] pair_ids (23,)
[loftr_wrapper.py] corres: (23, 5)
[2024-11-24 11:12:16.404] [warning] [FeatureManager.cpp:1589] start multi pair ransac GPU, pairs#=1
[2024-11-24 11:12:16.405] [warning] [FeatureManager.cpp:1695] after ransac, frame color_20241119_092326_327 and color_20241119_092324_286 has too few matches #0, ignore
> /home/themozel/uni/BundleSDF/bundlesdf.py(452)process_new_frame()
-> for id in ids:
(Pdb) 

[yiqiaoqingyuyu]

Hello, I’m facing the same issue. Have you managed to resolve it? I’m wondering if the author has provided any solutions.

[themozel]

Hi, No unfortunately there has been no response so far.

[yiqiaoqingyuyu]

Hi, No unfortunately there has been no response so far.

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.

[themozel]

Hi, No unfortunately there has been no response so far.

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.

What do you mean exactly? Are they discontinuous in their naming or the way they are captured (discontinuous frames).

[yiqiaoqingyuyu]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing.

I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset.

There could also be issues with non-consecutive frames, but I haven’t come across any so far.

[themozel]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing.

I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset.

There could also be issues with non-consecutive frames, but I haven’t come across any so far.

Thanks for pointing it out. Yeah I will check again but I think I made at the beginning that they correspond. Did that solve the issue on your side ?

[yiqiaoqingyuyu]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing.我的意思是你需要检查你的 RGB、蒙版和深度文件。虽然非连续帧的问题可能不会产生重大影响，但这些文件中的图像至少应一一对应。例如，RGB 和深度文件夹可能包含 1.png、2.png 和 3.png，但蒙版文件夹可能只有 1.png 和 3.png，缺少 2.png。 I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset.我最初遇到了同样的问题，因为我发现 mask 文件夹中的一些图像在我的数据集的 RGB 和 depth 文件夹中缺少相应的图像。 There could also be issues with non-consecutive frames, but I haven’t come across any so far.非连续帧也可能存在问题，但到目前为止我还没有遇到任何问题。

Thanks for pointing it out.谢谢你指出来。 Yeah I will check again but I think I made at the beginning that they correspond.是的，我会再次检查，但我想我在一开始就做了他们对应的工作。 Did that solve the issue on your side ?这解决了你这边的问题吗？

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing. I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset. There could also be issues with non-consecutive frames, but I haven’t come across any so far.

Thanks for pointing it out. Yeah I will check again but I think I made at the beginning that they correspond. Did that solve the issue on your side ?

Yes, it solved the issue on my side. I'm not sure what method you used to create your dataset, but I used XMem to generate the masks. This method was recommended by the author in previous discussions. Of course, there are many other methods available as well.https://github.com/NVlabs/BundleSDF/issues/35#issuecomment-1629630868

[themozel]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing.我的意思是你需要检查你的 RGB、蒙版和深度文件。虽然非连续帧的问题可能不会产生重大影响，但这些文件中的图像至少应一一对应。例如，RGB 和深度文件夹可能包含 1.png、2.png 和 3.png，但蒙版文件夹可能只有 1.png 和 3.png，缺少 2.png。 I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset.我最初遇到了同样的问题，因为我发现 mask 文件夹中的一些图像在我的数据集的 RGB 和 depth 文件夹中缺少相应的图像。 There could also be issues with non-consecutive frames, but I haven’t come across any so far.非连续帧也可能存在问题，但到目前为止我还没有遇到任何问题。

Thanks for pointing it out.谢谢你指出来。 Yeah I will check again but I think I made at the beginning that they correspond.是的，我会再次检查，但我想我在一开始就做了他们对应的工作。 Did that solve the issue on your side ?这解决了你这边的问题吗？

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing. I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset. There could also be issues with non-consecutive frames, but I haven’t come across any so far.

Thanks for pointing it out. Yeah I will check again but I think I made at the beginning that they correspond. Did that solve the issue on your side ?

Yes, it solved the issue on my side. I'm not sure what method you used to create your dataset, but I used XMem to generate the masks. This method was recommended by the author in previous discussions. Of course, there are many other methods available as well.https://github.com/NVlabs/BundleSDF/issues/35#issuecomment-1629630868

Great. Thanks for the info. I would then check my data more in details maybe I have missed something

[yiqiaoqingyuyu]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing. I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset. There could also be issues with non-consecutive frames, but I haven’t come across any so far.

Thanks for pointing it out. Yeah I will check again but I think I made at the beginning that they correspond. Did that solve the issue on your side ?

Of course, you should also check if your depth data has been scaled, and make sure the mask only has two pixel values, like [0, 1] or [0, 255], with 0 representing the background.

[yiqiaoqingyuyu]

Hi, No unfortunately there has been no response so far.嗨，不，很遗憾，到目前为止还没有任何回应。

I reviewed the depth, mask, and RGB images in my dataset and discovered that some depth maps and masks were discontinuous, making them unable to align with the corresponding RGB images. After ensuring the data was continuous during re-acquisition, the issue was resolved. I recommend checking your dataset to see if a similar problem exists.我查看了数据集中的深度、蒙版和 RGB 图像，发现一些深度图和蒙版是不连续的，导致它们无法与相应的 RGB 图像对齐。在确保数据在重新获取期间是连续的后，问题得到了解决。我建议检查您的数据集，看看是否存在类似的问题。

What do you mean exactly?你到底是什么意思？ Are they discontinuous in their naming or the way they are captured (discontinuous frames).它们的命名或捕获方式是否不连续（不连续帧）。

What I mean is that you need to check your RGB, mask, and depth files. While the issue of non-consecutive frames might not have a major impact, the images in these files should at least correspond one-to-one. For example, your RGB and depth folders might contain 1.png, 2.png, and 3.png, but your mask folder might only have 1.png and 3.png, with 2.png missing. I encountered the same issue initially because I found that some images in the mask folder were missing their corresponding images in the RGB and depth folders of my dataset. There could also be issues with non-consecutive frames, but I haven’t come across any so far.

Thanks for pointing it out. Yeah I will check again but I think I made at the beginning that they correspond. Did that solve the issue on your side ?

Of course, you should also check if your depth data has been scaled, and make sure the mask only has two pixel values, like [0, 1] or [0, 255], with 0 representing the background.

You can refer to this link for more details:https://github.com/NVlabs/BundleSDF/issues/142

[yiqiaoqingyuyu]

The solutions mentioned above are all based on issues with the dataset. If the dataset is correct, then I don't have a better solution for now.