search

NVlabs / BundleSDF

[CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
https://bundlesdf.github.io/
Other
1.08k stars 119 forks source link

[custom object capture] File "/home/mona/BundleSDF/nerf_runner.py", line 1195, in <listcomp> ret_list = [all_ret[k] for k in k_extract] KeyError: 'rgb_map' #113

Closed monajalal closed 1 year ago

monajalal commented 1 year ago

When I run the first command, it shows visualization of my object along with orientation but then I get this error towards the end. I have done depth align capture of depth+rgb following the same exact format (from output of $ identify --verbose your_img.png) using Intel RealSense 435 camera.

Please let me know if you know how to fix this bug or what has caused it. I am able to successfully run bundlesdf on the milk video.

(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/cup/ --out_folder /home/mona/BundleSDF/cup/out --use_segmenter 1 --use_gui 1 --debug_level 2

MESSAGES

[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  18394
Minimum depth: 0.0
Maximum depth: 65.464
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16992872555859441954
[bundlesdf.py] process frame 16992872555859441954
[bundlesdf.py] frame_pairs: 1
[loftr_wrapper.py] image0: torch.Size([1, 1, 400, 400])
[loftr_wrapper.py] net forward
[loftr_wrapper.py] mconf, 0.2013905793428421 0.9608154296875
[loftr_wrapper.py] pair_ids (444,)
[loftr_wrapper.py] corres: (444, 5)
[2023-11-06 11:06:30.533] [warning] [FeatureManager.cpp:1589] start multi pair ransac GPU, pairs#=1
[2023-11-06 11:06:30.534] [warning] [FeatureManager.cpp:1695] after ransac, frame 16992872555859441954 and 16992872240298557497 has too few matches #0, ignore
[bundlesdf.py] trying new ref frame 16992872240298557497
[bundlesdf.py] frame_pairs: 1
[2023-11-06 11:06:30.534] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (16992872555859441954, 16992872240298557497)
[bundlesdf.py] frame 16992872555859441954 has not suitable ref_frame, mark as FAIL
[2023-11-06 11:06:30.534] [warning] [Bundler.cpp:67] forgetting frame 16992872555859441954
[2023-11-06 11:06:30.534] [warning] [FeatureManager.cpp:469] forgetting frame 16992872555859441954
[bundlesdf.py] processNewFrame done 16992872555859441954
[bundlesdf.py] rematch_after_nerf: True
[2023-11-06 11:06:30.534] [warning] [Bundler.cpp:961] Welcome saveNewframeResult
[2023-11-06 11:06:30.553] [warning] [Bundler.cpp:1110] saveNewframeResult done
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  18567
Minimum depth: 0.0
Maximum depth: 65.464
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16992872556526650349
[bundlesdf.py] process frame 16992872556526650349
[bundlesdf.py] frame_pairs: 1
[loftr_wrapper.py] image0: torch.Size([1, 1, 400, 400])
[loftr_wrapper.py] net forward
[loftr_wrapper.py] mconf, 0.20056384801864624 0.9910906553268433
[loftr_wrapper.py] pair_ids (443,)
[loftr_wrapper.py] corres: (443, 5)
[2023-11-06 11:06:30.617] [warning] [FeatureManager.cpp:1589] start multi pair ransac GPU, pairs#=1
[2023-11-06 11:06:30.618] [warning] [FeatureManager.cpp:1695] after ransac, frame 16992872556526650349 and 16992872240298557497 has too few matches #0, ignore
[bundlesdf.py] trying new ref frame 16992872240298557497
[bundlesdf.py] frame_pairs: 1
[2023-11-06 11:06:30.618] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (16992872556526650349, 16992872240298557497)
[bundlesdf.py] frame 16992872556526650349 has not suitable ref_frame, mark as FAIL
[2023-11-06 11:06:30.618] [warning] [Bundler.cpp:67] forgetting frame 16992872556526650349
[2023-11-06 11:06:30.618] [warning] [FeatureManager.cpp:469] forgetting frame 16992872556526650349
[bundlesdf.py] processNewFrame done 16992872556526650349
[bundlesdf.py] rematch_after_nerf: True
[2023-11-06 11:06:30.618] [warning] [Bundler.cpp:961] Welcome saveNewframeResult
[2023-11-06 11:06:30.641] [warning] [Bundler.cpp:1110] saveNewframeResult done
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  17961
Minimum depth: 0.0
Maximum depth: 65.464
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16992872557193858743
[bundlesdf.py] process frame 16992872557193858743
[bundlesdf.py] frame_pairs: 1
[loftr_wrapper.py] image0: torch.Size([1, 1, 400, 400])
[loftr_wrapper.py] net forward
[loftr_wrapper.py] mconf, 0.20095179975032806 0.9904302358627319
[loftr_wrapper.py] pair_ids (446,)
[loftr_wrapper.py] corres: (446, 5)
[2023-11-06 11:06:30.705] [warning] [FeatureManager.cpp:1589] start multi pair ransac GPU, pairs#=1
[2023-11-06 11:06:30.705] [warning] [FeatureManager.cpp:1695] after ransac, frame 16992872557193858743 and 16992872240298557497 has too few matches #0, ignore
[bundlesdf.py] trying new ref frame 16992872240298557497
[bundlesdf.py] frame_pairs: 1
[2023-11-06 11:06:30.706] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (16992872557193858743, 16992872240298557497)
[bundlesdf.py] frame 16992872557193858743 has not suitable ref_frame, mark as FAIL
[2023-11-06 11:06:30.706] [warning] [Bundler.cpp:67] forgetting frame 16992872557193858743
[2023-11-06 11:06:30.706] [warning] [FeatureManager.cpp:469] forgetting frame 16992872557193858743
[bundlesdf.py] processNewFrame done 16992872557193858743
[bundlesdf.py] rematch_after_nerf: True
[2023-11-06 11:06:30.706] [warning] [Bundler.cpp:961] Welcome saveNewframeResult
[2023-11-06 11:06:30.724] [warning] [Bundler.cpp:1110] saveNewframeResult done
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  18336
Minimum depth: 0.0
Maximum depth: 65.464
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16992872557861067138
[bundlesdf.py] process frame 16992872557861067138
[bundlesdf.py] frame_pairs: 1
[loftr_wrapper.py] image0: torch.Size([1, 1, 400, 400])
[loftr_wrapper.py] net forward
[loftr_wrapper.py] mconf, 0.20050473511219025 0.9646672606468201
[loftr_wrapper.py] pair_ids (463,)
[loftr_wrapper.py] corres: (463, 5)
[2023-11-06 11:06:30.784] [warning] [FeatureManager.cpp:1589] start multi pair ransac GPU, pairs#=1
[2023-11-06 11:06:30.785] [warning] [FeatureManager.cpp:1695] after ransac, frame 16992872557861067138 and 16992872240298557497 has too few matches #0, ignore
[bundlesdf.py] trying new ref frame 16992872240298557497
[bundlesdf.py] frame_pairs: 1
[2023-11-06 11:06:30.785] [warning] [FeatureManager.cpp:2690] _raw_matches found exsting pair (16992872557861067138, 16992872240298557497)
[bundlesdf.py] frame 16992872557861067138 has not suitable ref_frame, mark as FAIL
[2023-11-06 11:06:30.785] [warning] [Bundler.cpp:67] forgetting frame 16992872557861067138
[2023-11-06 11:06:30.785] [warning] [FeatureManager.cpp:469] forgetting frame 16992872557861067138
[bundlesdf.py] processNewFrame done 16992872557861067138
[bundlesdf.py] rematch_after_nerf: True
[2023-11-06 11:06:30.785] [warning] [Bundler.cpp:961] Welcome saveNewframeResult
[2023-11-06 11:06:30.805] [warning] [Bundler.cpp:1110] saveNewframeResult done
[2023-11-06 11:06:34.347] [warning] [Bundler.cpp:49] Connected to nerf_port 9999
[2023-11-06 11:06:34.347] [warning] [FeatureManager.cpp:2084] Connected to port 5555
default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}}
[bundlesdf.py] last_stamp 16992872557861067138
[bundlesdf.py] keyframes#: 1
[tool.py] compute_scene_bounds_worker start
[tool.py] compute_scene_bounds_worker done
[tool.py] merge pcd
[tool.py] compute_translation_scales done
translation_cvcam=[  4.77210971  -6.71065795 -39.99999922], sc_factor=0.08181818891162776
[nerf_runner.py] Octree voxel dilate_radius:1
level 0, resolution: 16
level 1, resolution: 20
level 2, resolution: 24
level 3, resolution: 28
level 4, resolution: 34
level 5, resolution: 41
level 6, resolution: 49
level 7, resolution: 59
level 8, resolution: 71
level 9, resolution: 85
level 10, resolution: 102
level 11, resolution: 123
level 12, resolution: 148
level 13, resolution: 177
level 14, resolution: 213
level 15, resolution: 256
GridEncoder: input_dim=3 n_levels=16 level_dim=2 resolution=16 -> 256 per_level_scale=1.2030 params=(20411696, 2) gridtype=hash align_corners=False
sc_factor 0.08181818891162776
translation [  4.77210971  -6.71065795 -39.99999922]
[WARNING] batch has 0 intersections!!
[nerf_runner.py] denoise cloud
[nerf_runner.py] Denoising rays based on octree cloud
[nerf_runner.py] bad_mask#=0
rays torch.Size([0, 12])
Start training
[nerf_runner.py] train progress 0/2001
Traceback (most recent call last):
  File "run_custom.py", line 223, in <module>
    run_one_video(video_dir=args.video_dir, out_folder=args.out_folder, use_segmenter=args.use_segmenter, use_gui=args.use_gui)
  File "run_custom.py", line 107, in run_one_video
    run_one_video_global_nerf(out_folder=out_folder)
  File "run_custom.py", line 152, in run_one_video_global_nerf
    tracker.run_global_nerf(reader=reader, get_texture=True, tex_res=512)
  File "/home/mona/BundleSDF/bundlesdf.py", line 748, in run_global_nerf
    nerf.train()
  File "/home/mona/BundleSDF/nerf_runner.py", line 862, in train
    self.train_loop(batch.cuda())
  File "/home/mona/BundleSDF/nerf_runner.py", line 686, in train_loop
    rgb, extras = self.render(rays=batch, ray_ids=self.data_loader.batch_ray_ids, frame_ids=frame_ids,depth=target_d,lindisp=False,perturb=True,raw_noise_std=self.cfg['raw_noise_std'], near=batch[:,self.ray_near_slice], far=batch[:,self.ray_far_slice], get_normals=False)
  File "/home/mona/BundleSDF/nerf_runner.py", line 1195, in render
    ret_list = [all_ret[k] for k in k_extract]
  File "/home/mona/BundleSDF/nerf_runner.py", line 1195, in <listcomp>
    ret_list = [all_ret[k] for k in k_extract]
KeyError: 'rgb_map'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/spawn.py", line 126, in _main
    self = reduction.pickle.load(from_parent)
  File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/synchronize.py", line 110, in __setstate__
    self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
[2023-11-06 11:06:37.485] [warning] [Bundler.cpp:59] Destructor
[2023-11-06 11:06:37.486] [warning] [Bundler.cpp:59] Destructor

first frame: depth Screenshot from 2023-11-06 14-13-10

(base) mona@ada:~/BundleSDF/cup/depth$ identify -verbose 16992872240298557497.png
Image:
  Filename: 16992872240298557497.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: DirectClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: Gray
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    gray: 16-bit
  Channel statistics:
    Pixels: 307200
    Gray:
      min: 0  (0)
      max: 65464 (0.998917)
      mean: 29340.5 (0.447708)
      standard deviation: 19454.9 (0.296863)
      kurtosis: -1.13771
      skewness: -0.00858744
      entropy: 0.93454
  Colors: 256

color

(base) mona@ada:~/BundleSDF/cup/rgb$ identify -verbose 16992872240298557497.png
Image:
  Filename: 16992872240298557497.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: DirectClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: sRGB
  Type: TrueColor
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    red: 8-bit
    green: 8-bit
    blue: 8-bit
  Channel statistics:
    Pixels: 307200
    Red:
      min: 17  (0.0666667)
      max: 243 (0.952941)
      mean: 116.806 (0.458064)
      standard deviation: 25.8617 (0.101418)
      kurtosis: 1.06302
      skewness: -0.389394
      entropy: 0.813971
    Green:
      min: 14  (0.054902)
      max: 255 (1)
      mean: 120.094 (0.470955)
      standard deviation: 27.6778 (0.10854)
      kurtosis: 4.32311
      skewness: -1.4672
      entropy: 0.809217
    Blue:
      min: 9  (0.0352941)
      max: 255 (1)
      mean: 121.013 (0.47456)
      standard deviation: 29.8778 (0.117168)
      kurtosis: 3.6818
      skewness: -1.35518
      entropy: 0.819892
  Image statistics:
    Overall:
      min: 9  (0.0352941)
      max: 255 (1)
      mean: 119.304 (0.46786)
      standard deviation: 27.8058 (0.109042)
      kurtosis: 3.15316
      skewness: -1.11311
      entropy: 0.81436
  Rendering intent: Perceptual
  Gamma: 0.454545
  Chromaticity:
    red primary: (0.64,0.33)
    green primary: (0.3,0.6)
    blue primary: (0.15,0.06)
    white point: (0.3127,0.329)

Screenshot from 2023-11-06 14-13-22

mask Screenshot from 2023-11-06 14-13-35

(base) mona@ada:~/BundleSDF/cup/masks$ identify -verbose 16992872240298557497.png
Image:
  Filename: 16992872240298557497.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: PseudoClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: Gray
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 8-bit
  Channel depth:
    gray: 8-bit
  Channel statistics:
    Pixels: 307200
    Gray:
      min: 0  (0)
      max: 27 (0.105882)
      mean: 1.68495 (0.00660765)
      standard deviation: 6.53106 (0.025612)
      kurtosis: 11.0907
      skewness: 3.6181
      entropy: 0.336921
  Colors: 2
  Histogram:
    288029: (0,0,0) #000000 gray(0)
    19171: (27,27,27) #1B1B1B gray(27)
  Colormap entries: 256

MicrosoftTeams-image (2)

wenbowen123 commented 1 year ago

possible duplicate of https://github.com/NVlabs/BundleSDF/issues/115