search

NVlabs / BundleSDF

[CVPR 2023] BundleSDF: Neural 6-DoF Tracking and 3D Reconstruction of Unknown Objects
https://bundlesdf.github.io/
Other
992 stars 109 forks source link

[pcl::PLYWriter::writeASCII] Input point cloud has no data! [pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud! #116

Closed monajalal closed 9 months ago

monajalal commented 10 months ago

After using depth*1000.astype(uint16) and making sure they are in acceptable range, I am getting this new error:

(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/cup --out_folder /home/mona/BundleSDF/cup/out --use_segmenter 1 --use_gui 1 --debug_level 2
[2023-11-07 11:39:34.914] [warning] [Bundler.cpp:49] Connected to nerf_port 9999
[2023-11-07 11:39:34.914] [warning] [FeatureManager.cpp:2084] Connected to port 5555
default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}}
GUI started
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  3897
Minimum depth: 0.0
Maximum depth: 58.392
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16993034646708456934
[bundlesdf.py] process frame 16993034646708456934
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
Segmentation fault (core dumped)

Screenshot from 2023-11-07 14-44-10

I have this structure:

(base) mona@ada:~/BundleSDF/cup$ ls
total 148K
drwxrwxr-x 15 mona mona 4.0K Nov  7 13:51 ..
-rw-rw-r--  1 mona mona  225 Nov  7 13:53 cam_K.txt
drwxrwxr-x  2 mona mona  44K Nov  7 14:00 rgb
drwxrwxr-x  2 mona mona  44K Nov  7 14:00 depth
drwxrwxr-x  2 mona mona  36K Nov  7 14:01 masks
drwxrwxr-x  6 mona mona 4.0K Nov  7 14:02 .
drwxr-xr-x  3 root root 4.0K Nov  7 14:02 out

Screenshot from 2023-11-07 14-05-36

depth: Screenshot from 2023-11-07 14-06-01

mask:

Screenshot from 2023-11-07 14-41-51

depth map:

(base) mona@ada:~/BundleSDF/cup/depth$ identify -verbose 16993034646708456934.png 
Image:
  Filename: 16993034646708456934.png
  Format: PNG (Portable Network Graphics)
  Mime type: image/png
  Class: DirectClass
  Geometry: 640x480+0+0
  Units: Undefined
  Colorspace: Gray
  Type: Grayscale
  Base type: Undefined
  Endianness: Undefined
  Depth: 16-bit
  Channel depth:
    gray: 16-bit
  Channel statistics:
    Pixels: 307200
    Gray:
      min: 0  (0)
      max: 58392 (0.891005)
      mean: 9373.89 (0.143036)
      standard deviation: 21435.8 (0.327089)
      kurtosis: 1.42042
      skewness: 1.84944
      entropy: 0.635585
  Colors: 2
  Histogram:
    257884: (0,0,0) #000000000000 gray(0)
    49316: (58392,58392,58392) #E418E418E418 gray(89.1005%)
  Rendering intent: Undefined
  Gamma: 0.454545
  Background color: gray(255)
  Border color: gray(223)
  Matte color: gray(189)
  Transparent color: gray(0)
  Interlace: None
  Intensity: Undefined
  Compose: Over
  Page geometry: 640x480+0+0
  Dispose: Undefined
  Iterations: 0
  Compression: Zip
  Orientation: Undefined
  Properties:
    date:create: 2023-11-07T19:38:47+00:00
    date:modify: 2023-11-07T19:38:47+00:00
    png:IHDR.bit-depth-orig: 16
    png:IHDR.bit_depth: 16
    png:IHDR.color-type-orig: 0
    png:IHDR.color_type: 0 (Grayscale)
    png:IHDR.interlace_method: 0 (Not interlaced)
    png:IHDR.width,height: 640, 480
    signature: 40bf57548d02e85f7a5feac8ead3f2252dc9d1ae630004fff089a8dfee2e34c6
  Artifacts:
    filename: 16993034646708456934.png
    verbose: true
  Tainted: False
  Filesize: 11552B
  Number pixels: 307200
  Pixels per second: 77.7087MB
  User time: 0.000u
  Elapsed time: 0:01.003
  Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org

running it with debug level 4 didn't give me much different output:

(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/cup --out_folder /home/mona/BundleSDF/cup/out --use_segmenter 1 --use_gui 1 --debug_level 4
[2023-11-07 11:43:56.476] [warning] [Bundler.cpp:49] Connected to nerf_port 9999
[2023-11-07 11:43:56.476] [warning] [FeatureManager.cpp:2084] Connected to port 5555
default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}}
GUI started
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  3897
Minimum depth: 0.0
Maximum depth: 58.392
Minimum mask: 0
Maximum mask: 1
[bundlesdf.py] percentile denoise done
[bundlesdf.py] processNewFrame start 16993034646708456934
[bundlesdf.py] process frame 16993034646708456934
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
[pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud!
[pcl::PLYWriter::writeASCII] Input point cloud has no data!
Segmentation fault (core dumped)

I notice you mentioned in #87 to change the z-far. I am not sure where exactly to change it and to what value?

near: 0.1
far: 2

I saw this in config.yaml and looks like z-far, right? How to choose which value to change it to?

Here's my script for converting ros2 humble bags to images.

from pathlib import Path

from rosbags.highlevel import AnyReader
import numpy as np
from PIL import Image
from datetime import datetime
from matplotlib import image
import cv2
import os
import shutil

import matplotlib.pyplot as plt

depth_flag = False

if os.path.exists(os.path.join(os.getcwd(), 'depth')):
    shutil.rmtree(os.path.join(os.getcwd(), 'depth'))
    os.mkdir(os.path.join(os.getcwd(), 'depth'))
with AnyReader([Path('/home/mona/rosbag2_2023_11_06-15_44_24')]) as reader:
    connections = [x for x in reader.connections if x.topic == '/camera/camera/aligned_depth_to_color/image_raw']
    depth_flag = True

    for connection, timestamp, rawdata in reader.messages(connections=connections):
        msg = reader.deserialize(rawdata, connection.msgtype)
        timestamp_dt = datetime.fromtimestamp(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9)
        timestamp_str = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S.%f")

        timestamp_ns = msg.header.stamp.sec * 1e9 + msg.header.stamp.nanosec

        numeric_timestamp = int(timestamp_ns / 1e-9)

        w, h = msg.width, msg.height
        if rgb_flag:
            color_image_data = np.reshape(msg.data, (h, w, 3)) # for rgb image
            rgb_image_name = 'rgb/' + str(numeric_timestamp)[:20] + '.png'
            image = Image.fromarray(color_image_data)
            image.save(rgb_image_name)

        if depth_flag:

            depth_image_data = msg.data.reshape(h, w,-1) 

            depth = cv2.inRange(depth_image_data, 0.1, 2.0)
            depth_scaled = (depth*1000).astype(np.uint16)

            depth_image_name = 'depth/' + str(numeric_timestamp)[:20] + '.png'

            depth_data = np.array(depth_image_data, dtype=np.uint16)
            cv2.imwrite(depth_image_name, depth_scaled)

my min and max values across all depth images are: Minimum depth: 0 Maximum depth: 58392

when I run the milk data, here is the min and max values:

(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/milk/2022-11-18-15-10-24_milk --out_folder /home/mona/BundleSDF/milk/2022-11-18-15-10-24_milk/out --use_segmenter 1 --use_gui 1 --debug_level 4
[2023-11-07 12:47:41.999] [warning] [Bundler.cpp:49] Connected to nerf_port 9999
[2023-11-07 12:47:41.999] [warning] [FeatureManager.cpp:2084] Connected to port 5555
default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}}
GUI started
[bundlesdf.py] percentile denoise start
depth.shape (480, 640)
mask.shape (480, 640)
valid:  [[False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]
 ...
 [False False False ... False False False]
 [False False False ... False False False]
 [False False False ... False False False]]
valid.sum():  29741
Minimum depth: 0.0
Maximum depth: 0.917
Minimum mask: 0
Maximum mask: 1

when my custom data min and max values are:

valid.sum():  3897
Minimum depth: 0.0
Maximum depth: 0.58392

I would need to know how to set zfar value based on this information. I checked the git issues and it wasn't clear to me how this problem was resolved.

nuaayxy commented 9 months ago

I think its expecting some format of depth frame, make sure use uint16 and divide by 1e3 if needed