Difficulty in using Hololens Data

[saru-d2]

Hi,

First off, huge fan of this project. I want to use it with hololens2.

I'll quickly walk you through what I had tried.

So I parsed the data to the format that ios ARKIT provides, as that is what the demo data used. You can find the script used for the same here.

You can find the raw data here, and the parsed data here.

  0%|                                                                                                                        | 0/14 [00:00<?, ?it/s]/home/gridraster/saru/NeuralRecon/ops/back_project.py:29: UserWarning: This overload of nonzero is deprecated:
        nonzero(Tensor input, *, Tensor out)
Consider using one of the following signatures instead:
        nonzero(Tensor input, *, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629427478/work/torch/csrc/utils/python_arg_parser.cpp:766.)
  batch_ind = torch.nonzero(coords[:, 0] == batch).squeeze(1)
2021-08-03 21:02:14.444 | WARNING  | models.neucon_network:forward:184 - no valid points: scale 1
  7%|████████                                                                                                        | 1/14 [00:01<00:15,  1.18s/it]2021-08-03 21:02:14.737 | WARNING  | models.neucon_network:forward:184 - no valid points: scale 2
 50%|████████████████████████████████████████████████████████                                                        | 7/14 [00:03<00:02,  2.81it/s]2021-08-03 21:02:16.636 | WARNING  | models.neucon_network:forward:184 - no valid points: scale 2
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:05<00:00,  2.64it/s]

Summary:
    Total number of fragments: 14 
    Average keyframes/sec: 25.624008109191312
    Average GPU memory usage (GB): 0.7130301339285714 
    Max GPU memory usage (GB): 0.748046875 

I had tried running this with several datasets, and this is the only one that even provided a point cloud the rest, simply said

AssertionError: Reconstruction failed. Potential reasons could be:
                1. Wrong camera poses.
                2. Extremely difficult scene.
                If you can run with the demo data without any problem, please submit a issue with the failed data attatched, thanks!

hence why I'm putting this up.

Any help would be much appreciated, thanks!

Thanks, Saru

P.S. in the error message models.neucon_network:forward:184 - no valid points: scale 2 what does scale refer to?

[JiamingSuen]

I took a look at your data and I'm pretty sure that it is within the reconstruction capability of NeuralRecon. I recommend further checking the accuracy (and coordinate system) of the camera poses. Unfortunately, I don't have enough time recently to further debug your data.

P.S. in the error message models.neucon_network:forward:184 - no valid points: scale 2 what does scale refer to?

scale 2 refer to the coarse-to-fine level 2. no valid points means no voxel has a predicted occupancy score that is larger than the predefined threshold.

[HeCraneChen]

I took a look at your data and I'm pretty sure that it is within the reconstruction capability of NeuralRecon. I recommend further checking the accuracy (and coordinate system) of the camera poses. Unfortunately, I don't have enough time recently to further debug your data.

P.S. in the error message models.neucon_network:forward:184 - no valid points: scale 2 what does scale refer to?

scale 2 refer to the coarse-to-fine level 2. no valid points means no voxel has a predicted occupancy score that is larger than the predefined threshold.

Hi Jiaming, one followup question about this. I noticed that you set the threshold of occupancy as [0,0,0] currently. Have you tried more lenient threshold like a negative number? I'm wondering, is that gonna push up the completeness of NeuralRecon? Or, would doing that results in a trade-off, pushing up the completeness at the same time doing harm to the F-score?