How to compute the scale of omnidata depth?

[small-zeng]

Thanks for your work, but I do not find the code to compute the scale of omnidata depth in the paper.

[theNded]

This part isn't ready yet since we found it not quite stable. A simplified version is here by simply computing the median of the reprojected depth ratio between SfM and omnidata. I will revisit this part when I have some bandwidth. Thank you!

[small-zeng]

Thanks dor your reply, I have another question, when I have D435 depth which is noisy, Can I compute the scale between D435 depth and omnidata depth in this way ? Or by simply computing the median of thed depth ratio between D435 and omnidata ?

[theNded]

Or by simply computing the median of thed depth ratio between D435 and omnidata ?

I think this simplest practice would work fairly well, just be aware of the outliers from D435 (e.g. zero-depth pixels; pixels with depth > 5m) and compute the median.

[small-zeng]

Or by simply computing the median of thed depth ratio between D435 and omnidata ?

I think this simplest practice would work fairly well, just be aware of the outliers from D435 (e.g. zero-depth pixels; pixels with depth > 5m) and compute the median.

Thank you, I will try it

[small-zeng]

I have another question， Is this scale normal?

[theNded]

This looks incorrect. Realsense pixel depth by default uses mm (meaning that you need to divide pixel value by 1000). Some of the sensors have a depth factor of 2500 or 5000.

[small-zeng]

Thank you , I have divide 1000, but the ratio is still not unstable

[theNded]

OK, then would you please collect a set of images and compute the median scale per frame, and show the histogram of such medians?

[small-zeng]

The histogram is the ratio distribution of one depth image, but the variance is so large,  I cannot know what is wrong

---- Replied Message ----

From	Wei ***@***.***>
Date	08/30/2023 01:23
To	***@***.***>
Cc	Jing ***@***.***>、***@***.***>
Subject	Re: [theNded/torch-ash] How to compute the scale of omnidata depth? (Issue #23)

OK, then would you please collect a set of images and compute the median scale per frame, and show the histogram of such medians?

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: <theNded/torch-ash/issues/23/1697854040@github.com>

[theNded]

At a second glance I think it is not necessarily wrong. I can see the median is around 75 and the variance is ~50. This seems a reasonable distribution for monocular depth.

[small-zeng]

Thank you,I have another problem, obtain ing colmap pose cannot be online, so I use Intel t265 pose as input , Is your method sensitive to noise pose?

---- Replied Message ----

From	Wei ***@***.***>
Date	08/30/2023 01:35
To	***@***.***>
Cc	Jing ***@***.***>、***@***.***>
Subject	Re: [theNded/torch-ash] How to compute the scale of omnidata depth? (Issue #23)

At a second glance I think it is not necessarily wrong. I can see the median is around 75 and the variance is ~50. This seems a reasonable distribution for monocular depth.

—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: <theNded/torch-ash/issues/23/1697870197@github.com>

[small-zeng]

Hello，when I use sensor depth for scannet scene0050_00, the mesh is worse than learned depth，why is it?

[small-zeng]

When I load my data , I always occur in fusing dataset:

unordered_base::try_insert : Associated bucket and excess list full stdgpu::vector::size : Size out of bounds: -17 not in [0, 12182]. Clamping to 0 stdgpu::vector::size : Size out of bounds: -17 not in [0, 12182]. Clamping to 0 stdgpu::vector::size : Size out of bounds: -17 not in [0, 12182]. Clamping to 0 stdgpu::vector::size : Size out of bounds: -17 not in [0, 12182]. Clamping to 0 stdgpu::vector::pop_back : Object empty unordered_base::try_insert : Associated bucket and excess list full stdgpu::vector::size : Size out of bounds: -17 not in [0, 12182]. Clamping to 0 Fuse frame 31: 13%|████████████▊ | 31/239 [00:01<00:12, 16.03it/s] Traceback (most recent call last): File "demo/train_scene_recon.py", line 266, in <module> model.fuse_dataset(dataset, dilation) File "demo/train_scene_recon.py", line 63, in fuse_dataset fuser.fuse_dataset(dataset) File "/opt/anaconda3/envs/sdfstudio/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/mnt/dataset/zengjing/torch-ash/demo/rgbd_fusion.py", line 62, in fuse_dataset self.fuse_frame(datum) File "/opt/anaconda3/envs/sdfstudio/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/mnt/dataset/zengjing/torch-ash/demo/rgbd_fusion.py", line 227, in fuse_frame ) = self.grid.spatial_init_(points, dilation=self.dilation, bidirectional=True) File "/opt/anaconda3/envs/sdfstudio/lib/python3.8/site-packages/ash/grid.py", line 152, in spatial_init_ assert masks.all()

[theNded]

It looks like the hash map capacity is not enough. Would you please try to increase the capacity here https://github.com/theNded/torch-ash/blob/master/demo/train_scene_recon.py#L40?

I fixed this number for easier loading/saving, but perhaps I can put it in a metadata field for more flexibility. That will make the hash map size adaptive.

Another way to debug this is to visualize the mesh before the error (say extract the mesh at frame 25 and see how it looks like). This may help identify the issue.

[theNded]

Hello，when I use sensor depth for scannet scene0050_00, the mesh is worse than learned depth，why is it?

If you simply use the rgbd_fusion part without rescaling (which is the right way), the result should look better than this. If you apply the finetuning (i.e. the training part), then it is possible that the results look worse. You may want to add losses designed for RGB-D from this paper https://dazinovic.github.io/neural-rgbd-surface-reconstruction/.