Closed monajalal closed 9 months ago
After using depth*1000.astype(uint16) and making sure they are in acceptable range, I am getting this new error:
(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/cup --out_folder /home/mona/BundleSDF/cup/out --use_segmenter 1 --use_gui 1 --debug_level 2 [2023-11-07 11:39:34.914] [warning] [Bundler.cpp:49] Connected to nerf_port 9999 [2023-11-07 11:39:34.914] [warning] [FeatureManager.cpp:2084] Connected to port 5555 default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}} GUI started [bundlesdf.py] percentile denoise start depth.shape (480, 640) mask.shape (480, 640) valid: [[False False False ... False False False] [False False False ... False False False] [False False False ... False False False] ... [False False False ... False False False] [False False False ... False False False] [False False False ... False False False]] valid.sum(): 3897 Minimum depth: 0.0 Maximum depth: 58.392 Minimum mask: 0 Maximum mask: 1 [bundlesdf.py] percentile denoise done [bundlesdf.py] processNewFrame start 16993034646708456934 [bundlesdf.py] process frame 16993034646708456934 [pcl::PLYWriter::writeASCII] Input point cloud has no data! [pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud! [pcl::PLYWriter::writeASCII] Input point cloud has no data! Segmentation fault (core dumped)
I have this structure:
(base) mona@ada:~/BundleSDF/cup$ ls total 148K drwxrwxr-x 15 mona mona 4.0K Nov 7 13:51 .. -rw-rw-r-- 1 mona mona 225 Nov 7 13:53 cam_K.txt drwxrwxr-x 2 mona mona 44K Nov 7 14:00 rgb drwxrwxr-x 2 mona mona 44K Nov 7 14:00 depth drwxrwxr-x 2 mona mona 36K Nov 7 14:01 masks drwxrwxr-x 6 mona mona 4.0K Nov 7 14:02 . drwxr-xr-x 3 root root 4.0K Nov 7 14:02 out
depth:
mask:
depth map:
(base) mona@ada:~/BundleSDF/cup/depth$ identify -verbose 16993034646708456934.png Image: Filename: 16993034646708456934.png Format: PNG (Portable Network Graphics) Mime type: image/png Class: DirectClass Geometry: 640x480+0+0 Units: Undefined Colorspace: Gray Type: Grayscale Base type: Undefined Endianness: Undefined Depth: 16-bit Channel depth: gray: 16-bit Channel statistics: Pixels: 307200 Gray: min: 0 (0) max: 58392 (0.891005) mean: 9373.89 (0.143036) standard deviation: 21435.8 (0.327089) kurtosis: 1.42042 skewness: 1.84944 entropy: 0.635585 Colors: 2 Histogram: 257884: (0,0,0) #000000000000 gray(0) 49316: (58392,58392,58392) #E418E418E418 gray(89.1005%) Rendering intent: Undefined Gamma: 0.454545 Background color: gray(255) Border color: gray(223) Matte color: gray(189) Transparent color: gray(0) Interlace: None Intensity: Undefined Compose: Over Page geometry: 640x480+0+0 Dispose: Undefined Iterations: 0 Compression: Zip Orientation: Undefined Properties: date:create: 2023-11-07T19:38:47+00:00 date:modify: 2023-11-07T19:38:47+00:00 png:IHDR.bit-depth-orig: 16 png:IHDR.bit_depth: 16 png:IHDR.color-type-orig: 0 png:IHDR.color_type: 0 (Grayscale) png:IHDR.interlace_method: 0 (Not interlaced) png:IHDR.width,height: 640, 480 signature: 40bf57548d02e85f7a5feac8ead3f2252dc9d1ae630004fff089a8dfee2e34c6 Artifacts: filename: 16993034646708456934.png verbose: true Tainted: False Filesize: 11552B Number pixels: 307200 Pixels per second: 77.7087MB User time: 0.000u Elapsed time: 0:01.003 Version: ImageMagick 6.9.11-60 Q16 x86_64 2021-01-25 https://imagemagick.org
running it with debug level 4 didn't give me much different output:
(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/cup --out_folder /home/mona/BundleSDF/cup/out --use_segmenter 1 --use_gui 1 --debug_level 4 [2023-11-07 11:43:56.476] [warning] [Bundler.cpp:49] Connected to nerf_port 9999 [2023-11-07 11:43:56.476] [warning] [FeatureManager.cpp:2084] Connected to port 5555 default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}} GUI started [bundlesdf.py] percentile denoise start depth.shape (480, 640) mask.shape (480, 640) valid: [[False False False ... False False False] [False False False ... False False False] [False False False ... False False False] ... [False False False ... False False False] [False False False ... False False False] [False False False ... False False False]] valid.sum(): 3897 Minimum depth: 0.0 Maximum depth: 58.392 Minimum mask: 0 Maximum mask: 1 [bundlesdf.py] percentile denoise done [bundlesdf.py] processNewFrame start 16993034646708456934 [bundlesdf.py] process frame 16993034646708456934 [pcl::PLYWriter::writeASCII] Input point cloud has no data! [pcl::KdTreeFLANN::setInputCloud] Cannot create a KDTree with an empty input cloud! [pcl::PLYWriter::writeASCII] Input point cloud has no data! Segmentation fault (core dumped)
I notice you mentioned in #87 to change the z-far. I am not sure where exactly to change it and to what value?
near: 0.1 far: 2
I saw this in config.yaml and looks like z-far, right? How to choose which value to change it to?
Here's my script for converting ros2 humble bags to images.
ros2 humble bags
images
from pathlib import Path from rosbags.highlevel import AnyReader import numpy as np from PIL import Image from datetime import datetime from matplotlib import image import cv2 import os import shutil import matplotlib.pyplot as plt depth_flag = False if os.path.exists(os.path.join(os.getcwd(), 'depth')): shutil.rmtree(os.path.join(os.getcwd(), 'depth')) os.mkdir(os.path.join(os.getcwd(), 'depth')) with AnyReader([Path('/home/mona/rosbag2_2023_11_06-15_44_24')]) as reader: connections = [x for x in reader.connections if x.topic == '/camera/camera/aligned_depth_to_color/image_raw'] depth_flag = True for connection, timestamp, rawdata in reader.messages(connections=connections): msg = reader.deserialize(rawdata, connection.msgtype) timestamp_dt = datetime.fromtimestamp(msg.header.stamp.sec + msg.header.stamp.nanosec * 1e-9) timestamp_str = timestamp_dt.strftime("%Y-%m-%d %H:%M:%S.%f") timestamp_ns = msg.header.stamp.sec * 1e9 + msg.header.stamp.nanosec numeric_timestamp = int(timestamp_ns / 1e-9) w, h = msg.width, msg.height if rgb_flag: color_image_data = np.reshape(msg.data, (h, w, 3)) # for rgb image rgb_image_name = 'rgb/' + str(numeric_timestamp)[:20] + '.png' image = Image.fromarray(color_image_data) image.save(rgb_image_name) if depth_flag: depth_image_data = msg.data.reshape(h, w,-1) depth = cv2.inRange(depth_image_data, 0.1, 2.0) depth_scaled = (depth*1000).astype(np.uint16) depth_image_name = 'depth/' + str(numeric_timestamp)[:20] + '.png' depth_data = np.array(depth_image_data, dtype=np.uint16) cv2.imwrite(depth_image_name, depth_scaled)
my min and max values across all depth images are: Minimum depth: 0 Maximum depth: 58392
when I run the milk data, here is the min and max values:
(py38) root@ada:/home/mona/BundleSDF# python run_custom.py --mode run_video --video_dir /home/mona/BundleSDF/milk/2022-11-18-15-10-24_milk --out_folder /home/mona/BundleSDF/milk/2022-11-18-15-10-24_milk/out --use_segmenter 1 --use_gui 1 --debug_level 4 [2023-11-07 12:47:41.999] [warning] [Bundler.cpp:49] Connected to nerf_port 9999 [2023-11-07 12:47:41.999] [warning] [FeatureManager.cpp:2084] Connected to port 5555 default_cfg {'backbone_type': 'ResNetFPN', 'resolution': (8, 2), 'fine_window_size': 5, 'fine_concat_coarse_feat': True, 'resnetfpn': {'initial_dim': 128, 'block_dims': [128, 196, 256]}, 'coarse': {'d_model': 256, 'd_ffn': 256, 'nhead': 8, 'layer_names': ['self', 'cross', 'self', 'cross', 'self', 'cross', 'self', 'cross'], 'attention': 'linear', 'temp_bug_fix': False}, 'match_coarse': {'thr': 0.2, 'border_rm': 2, 'match_type': 'dual_softmax', 'dsmax_temperature': 0.1, 'skh_iters': 3, 'skh_init_bin_score': 1.0, 'skh_prefilter': True, 'train_coarse_percent': 0.4, 'train_pad_num_gt_min': 200}, 'fine': {'d_model': 128, 'd_ffn': 128, 'nhead': 8, 'layer_names': ['self', 'cross'], 'attention': 'linear'}} GUI started [bundlesdf.py] percentile denoise start depth.shape (480, 640) mask.shape (480, 640) valid: [[False False False ... False False False] [False False False ... False False False] [False False False ... False False False] ... [False False False ... False False False] [False False False ... False False False] [False False False ... False False False]] valid.sum(): 29741 Minimum depth: 0.0 Maximum depth: 0.917 Minimum mask: 0 Maximum mask: 1
when my custom data min and max values are:
valid.sum(): 3897 Minimum depth: 0.0 Maximum depth: 0.58392
I would need to know how to set zfar value based on this information. I checked the git issues and it wasn't clear to me how this problem was resolved.
I think its expecting some format of depth frame, make sure use uint16 and divide by 1e3 if needed
After using depth*1000.astype(uint16) and making sure they are in acceptable range, I am getting this new error:
I have this structure:
depth:![Screenshot from 2023-11-07 14-06-01](https://github.com/NVlabs/BundleSDF/assets/1892917/44c8e393-1536-4fdd-80b9-09c12e2120c2)
mask:
depth map:
running it with debug level 4 didn't give me much different output:
I notice you mentioned in #87 to change the z-far. I am not sure where exactly to change it and to what value?
I saw this in config.yaml and looks like z-far, right? How to choose which value to change it to?
Here's my script for converting
ros2 humble bags
toimages
.my min and max values across all depth images are: Minimum depth: 0 Maximum depth: 58392
when I run the milk data, here is the min and max values:
when my custom data min and max values are:
I would need to know how to set zfar value based on this information. I checked the git issues and it wasn't clear to me how this problem was resolved.