search

hovsg / HOV-SG

[RSS2024] Official implementation of "Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation"
https://hovsg.github.io
MIT License
186 stars 12 forks source link

Run HOV-SG on own datassets #14

Open ADetilie opened 2 months ago

ADetilie commented 2 months ago

I want to run HOV-SG on own dataset.

I controlled simulated robot and gathered RGB images, depth images and pos. Than - reformatted this data in the structure, similar to hm3dsem dataset. I've created 2 different datasets:

  1. with low-res RGB and depth images (640x480) and
  2. high-res RGB and depth images (1080x720)

HOV-SG failed to build a proper Scene Graphs on those datasets (I will post full logs and results for each dataset in comments to this issue).

Command I've used for graph generation:

python application/create_graph.py main.dataset=<path to my dataset> main.save_path=<output path>

Config:

main:
  device: cuda
  dataset: hm3dsem # scannet, replica
  scene_id: 00824-Dd4bFSTQ8gi # scene0011_00
  dataset_path: /data/buechner/hm3dsem_walks/val/00824-Dd4bFSTQ8gi
  save_path: /data/buechner/hovsg/
models:
  clip:
    type:  ViT-H-14 # ViT-L/14@336px # ViT-H-14
    checkpoint: checkpoints/laion2b_s32b_b79k.bin 
    # checkpoint: checkpoints/ovseg_clipl14_9a1909.pth checkpoints/laion2b_s32b_b79k.bin
  sam:
    checkpoint: checkpoints/sam_vit_h_4b8939.pth
    type: vit_h
    points_per_side: 12
    pred_iou_thresh: 0.88
    points_per_batch: 144
    crop_n_layers: 0
    stability_score_thresh: 0.95
    min_mask_region_area: 100
pipeline:
  create_graph: True
  voxel_size: 0.02
  skip_frames: 10
  init_overlap_thresh: 0.75
  overlap_thresh_factor: 0.025
  iou_thresh: 0.05
  clip_masked_weight: 0.4418
  clip_bbox_margin: 50 # in pixels
  feature_dbscan_eps: 0.01
  max_mask_distance: 10000 # 6.4239 in meters
  min_pcd_points: 100
  depth_weighting: false
  grid_resolution: 0.05
  merge_type: hierarchical # hierarchical, sequential
  save_intermediate_results: true
  obj_labels: HM3DSEM_LABELS
  merge_objects_graph: false

Could you help identifying possible problems and give some hints how I can debug the issue? Details will be in comments to this issue.

Data examples.

RGB: 27 000001408

Depth: 27 000001408 (depth)

Pos: 27.000001408.txt

ADetilie commented 2 months ago

Logs that I am getting while running HOV-SG on high-res dataset. Scene Graph generation crashes and there is no way to visualize it. But there is no any WARNINGS during generation.

(hovsg_exp) ➜  HOV-SG git:(main) ✗ python application/create_graph.py main.dataset=hm3dsem main.dataset_path=/home/dev/repos/hov-sg-dataset-tools/hov-sg-dataset-tools/hov_sg_dataset_tools/output_highres/sim_office_dataset_highres/val main.save_path=data/sim_office_dataset_highres
[2024-08-07 20:04:35,523][root][INFO] - Loaded ViT-H-14 model config.
[2024-08-07 20:04:43,524][root][INFO] - Loading pretrained ViT-H-14 weights (checkpoints/laion2b_s32b_b79k.bin).
Creating RGB-D point cloud: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 358/358 [00:36<00:00,  9.79it/s]
Extracting features: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 358/358 [38:34<00:00,  6.47s/it]
Merging 3d masks hierarchically
  0%|                                                                                                                                                                               | 0/179 [00:00<?, ?it/s]/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:507: RuntimeWarning: invalid value encountered in scalar divide
  obj_1_overlap = overlap_volume / bbox1_volume
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:508: RuntimeWarning: invalid value encountered in scalar divide
  obj_2_overlap = overlap_volume / bbox2_volume
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:511: RuntimeWarning: invalid value encountered in scalar divide
  iou = overlap_volume / (bbox1_volume + bbox2_volume - overlap_volume)
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 179/179 [00:08<00:00, 20.59it/s]
th:  0.7251404494382022
 99%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ | 89/90 [00:11<00:00,  7.83it/s]
th:  0.7004213483146067
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [00:21<00:00,  2.05it/s]
th:  0.6759895301327885
 96%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▋       | 22/23 [00:41<00:01,  1.89s/it]
th:  0.6521258937691521
 92%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████              | 11/12 [01:22<00:07,  7.47s/it]
th:  0.6293986210418794
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [02:41<00:00, 26.84s/it]
th:  0.6093986210418794
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [05:12<00:00, 104.30s/it]
th:  0.5968986210418794
 50%|████████████████████████████████████████████████████████████████████████████████████                                                                                    | 1/2 [08:40<08:40, 520.20s/it]
th:  0.5968986210418794
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [15:04<00:00, 904.41s/it]
Fusing features: 4702it [00:04, 1151.07it/s]
number of masks:  4702
number of pcds in hovsg:  4702
-- removing small and empty masks --
number of masked pcds:  2351
number of mask_feats:  2351
masked pcds saved to disk in data/sim_office_dataset_highres/hm3dsem/00824-Dd4bFSTQ8gi
full pcd saved to disk in data/sim_office_dataset_highres/hm3dsem/00824-Dd4bFSTQ8gi
full pcd feats saved to disk in data/sim_office_dataset_highres/hm3dsem/00824-Dd4bFSTQ8gi
hm3dsem
segmenting floors...
downpcd (452, 3)
bins 933.3560382305654
distance 20.0
0.2647374062165059
min_peak_height 1.0
clustred_peaks [-6.80846019 -5.17783817 -4.57760921  0.49432553]
floors [[-6.808460189483645, -5.177838171996163], [-4.577609208503837, 0.4943255330063092]]
number of floors:  2
segmenting rooms...
grid_size:  0.05
occupancy_map shape:  (40, 40)
range of dist:  0.0 5.656854
number of seeds, aka rooms:  1
area of seed 0:  155.5
min_area:  100.0
number of contours after remove small seeds:  1
room_vertices shape:  (2, 1244)
Assign floor points to rooms: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 354.37it/s]
Computing room features: 358it [00:18, 19.47it/s]
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:95: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(room_2d_points[:, 0], room_2d_points[:, 1], s=0.1, c=cmap(room_idx))
assign camera to room:   0%|                                                                                                                                                        | 0/358 [00:00<?, ?it/s]/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:112: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(pos[0], pos[1], s=3.0, c=pose_cmap(closest_room_idx))
assign camera to room: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 358/358 [00:00<00:00, 2842.38it/s]
[0 1 2 3 4 5 6 7 8 9]
number of rooms in floor 0 is 2
grid_size:  0.05
occupancy_map shape:  (80, 480)
range of dist:  0.0 5.8309517
number of seeds, aka rooms:  9
area of seed 0:  250.0
area of seed 1:  47.0
area of seed 2:  69.5
area of seed 3:  81.0
area of seed 4:  105.5
area of seed 5:  0.0
area of seed 6:  114.5
area of seed 7:  143.5
area of seed 8:  88.0
min_area:  100.0
number of contours after remove small seeds:  4
room_vertices shape:  (4, 2)
Assign floor points to rooms: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  5.06it/s]
Computing room features: 358it [00:17, 20.43it/s]
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:95: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(room_2d_points[:, 0], room_2d_points[:, 1], s=0.1, c=cmap(room_idx))
assign camera to room:   0%|                                                                                                                                                        | 0/358 [00:00<?, ?it/s]/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:112: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(pos[0], pos[1], s=3.0, c=pose_cmap(closest_room_idx))
assign camera to room: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 358/358 [00:00<00:00, 1782.93it/s]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
number of rooms in floor 1 is 4
segmenting/identifying objects...
Floor: 0:   0%|                                                                                                                                                                       | 0/2 [00:00<?, ?it/s/home/dev/repos/HOV-SG/hovsg/graph/graph.py:647: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). Consider using `matplotlib.pyplot.close()`.
  fig, ax = plt.subplots()
Floor: 1: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [02:55<00:00, 87.60s/it]
number of objects:  2010                                                                                                                                                                                    
Creating graph...
reverse pose sequence
Error executing job with overrides: ['main.dataset=hm3dsem', 'main.dataset_path=/home/dev/repos/hov-sg-dataset-tools/hov-sg-dataset-tools/hov_sg_dataset_tools/output_highres/sim_office_dataset_highres/val', 'main.save_path=data/sim_office_dataset_highres']
Traceback (most recent call last):
  File "/home/dev/repos/HOV-SG/application/create_graph.py", line 34, in main
    hovsg.build_graph(save_path=save_dir)
  File "/home/dev/repos/HOV-SG/hovsg/graph/graph.py", line 832, in build_graph
    self.create_nav_graph()
  File "/home/dev/repos/HOV-SG/hovsg/graph/graph.py", line 866, in create_nav_graph
    sparse_stairs_voronoi = nav_graph.get_stairs_graph_with_poses_v2(
  File "/home/dev/repos/HOV-SG/hovsg/graph/navigation_graph.py", line 844, in get_stairs_graph_with_poses_v2
    min_height = np.min([pos[1] for pos in tar_poses])
  File "/home/dev/miniforge3/envs/hovsg_exp/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 2953, in min
    return _wrapreduction(a, np.minimum, 'min', axis, None, out,
  File "/home/dev/miniforge3/envs/hovsg_exp/lib/python3.9/site-packages/numpy/core/fromnumeric.py", line 88, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
ValueError: zero-size array to reduction operation minimum which has no identity

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
ADetilie commented 2 months ago

Logs that I am getting while running HOV-SG on high-res dataset. Scene graph generation succeeds. But there are a lot of Warnings like the following:

[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.

Logs:

(hovsg_exp) ➜  HOV-SG git:(main) ✗ python application/create_graph.py main.dataset=hm3dsem main.dataset_path=/home/dev/repos/hov-sg-dataset-tools/hov-sg-dataset-tools/hov_sg_dataset_tools/output/sim_office_dataset/val main.save_path=data/sim_office_dataset_2
[2024-08-07 16:43:30,059][root][INFO] - Loaded ViT-H-14 model config.
[2024-08-07 16:43:37,954][root][INFO] - Loading pretrained ViT-H-14 weights (checkpoints/laion2b_s32b_b79k.bin).
Creating RGB-D point cloud: 100%|█████████████| 266/266 [00:11<00:00, 23.53it/s]
Extracting features:  58%|███████████▌        | 153/266 [09:19<07:48,  4.15s/it]/home/dev/repos/HOV-SG/hovsg/dataloader/generic.py:121: RuntimeWarning: Mean of empty slice.
  if Z.mean() > filter_distance:
/home/dev/miniforge3/envs/hovsg_exp/lib/python3.9/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in divide
  ret = ret.dtype.type(ret / rcount)
Extracting features: 100%|████████████████████| 266/266 [16:41<00:00,  3.77s/it]
Merging 3d masks hierarchically
  0%|                                                                                                                                        | 0/133 [00:00<?, ?it/s]/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:507: RuntimeWarning: invalid value encountered in scalar divide
  obj_1_overlap = overlap_volume / bbox1_volume
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:508: RuntimeWarning: invalid value encountered in scalar divide
  obj_2_overlap = overlap_volume / bbox2_volume
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:511: RuntimeWarning: invalid value encountered in scalar divide
  iou = overlap_volume / (bbox1_volume + bbox2_volume - overlap_volume)
 57%|████████████████████████████████████████████████████████████████████████▌                                                      | 76/133 [00:02<00:01, 39.09it/s][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 133/133 [00:05<00:00, 26.46it/s]
th:  0.725189393939394
 55%|██████████████████████████████████████████████████████████████████████▋                                                         | 37/67 [00:02<00:02, 12.69it/s][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 99%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████  | 66/67 [00:05<00:00, 11.00it/s]
th:  0.7005681818181818
 56%|███████████████████████████████████████████████████████████████████████▌                                                        | 19/34 [00:04<00:04,  3.29it/s][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 97%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   | 33/34 [00:11<00:00,  2.99it/s]
th:  0.6763257575757575
 53%|████████████████████████████████████████████████████████████████████▎                                                            | 9/17 [00:08<00:08,  1.10s/it][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:21<00:00,  1.26s/it]
th:  0.6528882575757575
 44%|█████████████████████████████████████████████████████████▊                                                                        | 4/9 [00:14<00:18,  3.67s/it][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 89%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████▌              | 8/9 [00:41<00:05,  5.18s/it]
th:  0.6310132575757575
 40%|████████████████████████████████████████████████████                                                                              | 2/5 [00:27<00:42, 14.15s/it][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 80%|████████████████████████████████████████████████████████████████████████████████████████████████████████                          | 4/5 [01:20<00:20, 20.16s/it]
th:  0.6122632575757575
 33%|███████████████████████████████████████████▎                                                                                      | 1/3 [00:55<01:50, 55.34s/it][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 67%|██████████████████████████████████████████████████████████████████████████████████████▋                                           | 2/3 [02:39<01:19, 79.56s/it]
th:  0.5997632575757575
  0%|                                                                                                                                                            | 0/2 [00:00<?, ?it/s][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
 50%|█████████████████████████████████████████████████████████████████████████▌                                                                         | 1/2 [05:08<05:08, 308.56s/it]
th:  0.5997632575757575
  0%|                                                                                                                                                            | 0/1 [00:00<?, ?it/s][Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [05:32<00:00, 332.66s/it]
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.
Fusing features: 2843it [00:02, 1146.37it/s]
number of masks:  2843
number of pcds in hovsg:  2843
-- removing small and empty masks --
number of masked pcds:  1420
number of mask_feats:  1420
masked pcds saved to disk in data/sim_office_dataset_2/hm3dsem/00824-Dd4bFSTQ8gi
full pcd saved to disk in data/sim_office_dataset_2/hm3dsem/00824-Dd4bFSTQ8gi
full pcd feats saved to disk in data/sim_office_dataset_2/hm3dsem/00824-Dd4bFSTQ8gi
hm3dsem
segmenting floors...
downpcd (304, 3)
bins 770.5133703062577
distance 20.0
0.21038961038961038
min_peak_height 1.0
clustred_peaks [-6.16748429  0.13671601]
floors [[-6.167484293790128, 0.1367160087156165]]
number of floors:  1
segmenting rooms...
grid_size:  0.05
occupancy_map shape:  (80, 500)
range of dist:  0.0 5.3851647
number of seeds, aka rooms:  8
area of seed 0:  499.5
area of seed 1:  55.0
area of seed 2:  158.5
area of seed 3:  187.0
area of seed 4:  214.0
area of seed 5:  12.0
area of seed 6:  237.0
area of seed 7:  122.0
min_area:  100.0
number of contours after remove small seeds:  6
room_vertices shape:  (6, 2)
Assign floor points to rooms: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:01<00:00,  5.19it/s]
Computing room features: 266it [00:12, 21.10it/s]
/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:95: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(room_2d_points[:, 0], room_2d_points[:, 1], s=0.1, c=cmap(room_idx))
assign camera to room:   0%|                                                                                                                                   | 0/266 [00:00<?, ?it/s]/home/dev/repos/HOV-SG/hovsg/utils/graph_utils.py:112: UserWarning: *c* argument looks like a single numeric RGB or RGBA sequence, which should be avoided as value-mapping will have precedence in case its length matches with *x* & *y*.  Please use the *color* keyword-argument or provide a 2D array with a single row if you intend to specify the same RGB or RGBA value for all points.
  plt.scatter(pos[0], pos[1], s=3.0, c=pose_cmap(closest_room_idx))
assign camera to room: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 266/266 [00:00<00:00, 785.61it/s]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
[0 1 2 3 4 5 6 7 8 9]
number of rooms in floor 0 is 6
segmenting/identifying objects...
Floor: 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:31<00:00, 31.89s/it]
number of objects:  1391                                                                                                                                                               
Creating graph...
reverse pose sequence
Getting paths between all nodes. Node number: 54/107
time for computing all pairs shortest path:  0.02669239044189453  seconds
Sparsifying graph: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 54/54 [00:00<00:00, 1934.49it/s]
connecting stairs and floor 0                                                                                                                                                          
(array([-6.62589794, -6.60962695, -6.5992234 , -6.59856338, -6.59524372,
       -6.43024765, -6.38059824, -6.36498715, -6.36228034, -6.31968969,
       -6.31896103, -6.23970005, -6.0914886 , -6.01070847, -6.00251975,
       -5.99066327, -5.98368079, -5.97462351, -5.97058707, -5.96709587,
       -5.96069087, -5.95825188, -5.94611777, -5.94327459, -5.93977536,
       -5.93885587]), array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1]))
(array([-6.62589794]), array([27]))
stairs, floor
(44.77137158179833, 470.0661705966118, 0) (50.5, 484.5, 0)
(13.62252916472627, -6.002519752903365, 1.0946620303565626) (14.055544046827917, -6.625897936782326, 1.2665208829026127)
0 0
number of floors:  1
number of rooms:  6
number of objects:  1391
HOV-SG is built successfully

BUT when I am visualizeing graph, I am getting image like this (graph is empty):

Screenshot 2024-08-08 at 19 30 16
Tom-Huang commented 2 months ago 
[Open3D WARNING] The number of points is 0 when creating axis-aligned bounding box.
[Open3D WARNING] [KDTreeFlann::SetRawData] Failed due to no data.

It seems that the point cloud is empty. You could print the point cloud at this line to see what happened (insert print(frames_pcd)).

I also notice that you are using the main.dataset: hm3dsem option in the config, meaning that the data structure and coordinate system of your data are the same as habitat3DSem. You should make sure that the horizontal field of view of your camera is 90 degrees (the resolution doesn't matter for now), and the Y axis of the global coordinate frame is pointing upward (we rely on point clouds's Y coordinates to segment floors and rooms).

mga-97 commented 2 months ago

I also had a similar issue, in my case the problem was to do with the scaling of the depth, judging by the depth image posted here it could be the same. The depth array should be scaled by 1000.0 before saving and then in HOV-SG it will be scaled back down. If you haven't done the scaling before saving the depth PNG you should set the scale factor to 1.0, although you should probably do the scaling so you don't lose depth precision otherwise you are saving the depths as integers.

Tom-Huang commented 2 months ago

@mga-97 In our dataset, since we saved the depth image as a .npy file, we didn't scale the depth before saving. That is to say, the depth value at each pixel location is meter value and is floating point value.

mga-97 commented 2 months ago

@Tom-Huang Are you referring to the real world data you recorded? It seemed to me from the README that for "main.dataset: hm3dsem" the depth is PNG and the scaling used is here

Tom-Huang commented 2 months ago

@mga-97 ah, you are right. I didn't notice the refactoring.

Benaidandan commented 6 days ago

Are you successful now?