Problem with sens's alignment

[Maple-3d]

Hi @Sekunde , Thanks for your contribution！we were able to get the demo set up and running nicely!But when we tested our own RGBD data, we encountered a problem.We successfully generated the .scene files, but the final result is strange, it seems that the whole is misaligned. Did you use the Alignment in the Scannet code to align the scene when you test?

[Sekunde]

Hi, @Maple-3d

How is the misalignment looks like? is that rotation or translation misaligned or flipped?

we use this piece of code to transform the prediction results (in voxel space) to mesh (world) space; https://github.com/Sekunde/3D-SIS/blob/master/tools/scannet_benchmark/vox2mesh.py#L133-L142 where we export the world2grid and save it in the .scene file; basically for your own rgbd data, you need save the world2grid matrix in the .scene file.

[Maple-3d]

Hi!Thanks!@Sekunde The problem seems like a kind of rotation, there are lots of boxes but all in the wrong places,and some random point clouds are colored. By the way, in the visiualization step, a txt file miss in the scans folder.I copy yours in scans folder in 'example result' folder,and I change the axisAlignment matrix to the basic, feel like the problem from. How did you generate the txt file?Or how to generate the axisAlignment matrix?

[Sekunde]

can you share a screenshot or so? And I would suggest visualizing the voxel space first to make sure the problem lies in the transformation from voxel space to mesh space.

https://github.com/Sekunde/3D-SIS/blob/108ef46e5c3be71b26824914a7e28537b9cdf3c2/tools/scannet_benchmark/vox2mesh.py#L144

this line of code will save the predicted box and mask maybe you can visualize them first, or if you can pass me the folder of the prediction; maybe I can have a look.

[Maple-3d]

scene0529_00_misalign.zip Glad to hear from you!! The screenshot is this.And the predicted folder is this. misaligned_data.zip By the way, the scene0529_00.txt in scans folder was made by myself. The axisMatrix may be wrong,I don't know how to generate this scene0529_00.txt.Please help!

[Sekunde]

I have a look on your data, scene0529_00.txt may not be the reason as we do not use it in the demo code; but i do notice a different world2grid.txt file.

world2grid.txt

can you use this world2grid.txt to replace yours in images_square folder?

[Maple-3d]

Hi！Thanks！ I have tried the world2grid.txt that you send. But the program can't run with the error :

`loaded mode from ../checkpoints/ScanNet/benchmark/checkpoints/step_1205541.pth starting test on whole scan.... 0%| | 0/1 [00:00<?, ?it/s]error: no valid image indices error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: no valid depths error: no valid image indices error: nothing in frustum bounds error: no valid image indices error: nothing in frustum bounds error: nothing in frustum bounds error: nothing in frustum bounds error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths error: no valid depths ['toilet_full/scannet_chunk/scene0529_000.scene']: (invalid sample: no valid projection)

Traceback (most recent call last): File "main.py", line 84, in benchmark(args) File "/aiLab2/ZHW/3D-SIS-master/3D-SIS/3D-SIS/lib/model/trainval.py", line 73, in benchmark SolverWrapper.benchmark(net, dataloader, logger) File "/aiLab2/ZHW/3D-SIS-master/3D-SIS/3D-SIS/lib/model/trainval.py", line 687, in benchmark net.forward(blobs, 'TEST', killing_inds) File "/aiLab2/ZHW/3D-SIS-master/3D-SIS/3D-SIS/lib/nets/network.py", line 237, in forward imageft = imageft.view(sz[0], sz[1], sz[2], sz[3], self.batch_size) UnboundLocalError: local variable 'sz' referenced before assignment

If you suspect this is an IPython 7.16.1 bug, please report it at: https://github.com/ipython/ipython/issues or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via: %config Application.verbose_crash=True`

Do you know how to fix this?It seems like the depth data can't match the new world2grid.txt.

[Sekunde]

Yeah, I took the world2grid.txt from the data I generated, it seems your data seems a bit different; the problem is I don't have too much time now (need to prepare some interviews), but I will try to squeeze in some time recently to check on this, please stay tuned.

[Maple-3d]

Thanks really! I will be waiting for your messgae.

[Maple-3d]

Hello, I am still working on your paper recently. I would like to ask if your own scene was collected through BundleFusion, and is there any configuration item to adjust the alignment axis?

[Sekunde]

@Maple-3d

Hi, sry for coming back a bit late.

I actually run an inference on your data; and also get bad results. But I do notice something different.

In order to track down what is the problem, I first visualize the results in voxel space and there is already something wrong. (see the screenshot) our data is axis-aligned and Y-axis UP whereas your data is not axis-aligned and z-up; For aligning data, you can use the following:

New axis alignments : for all 1513 original ScanNet scans, as axisAlignment in the .txt info file (a 4x4 matrix encoding the rigid transform to axis alignment for the scan, as 16 values in row-major order).

reference: http://www.scan-net.org/changelog#scannet-v2-2018-06-11

[Maple-3d]

Thank you for replying to my message in your busy schedule！！！I will try this!

[Maple-3d]

Hi！@Sekunde I check your example data,and i found your ply file is Z-axis UP.This does not seem to be the problem from. I change my data to Y-axis, the phenomenen is different.The misalignment disappear and the box bounds wrong.

And I check my benchmark result, it seems like something wrong, too much "0" type in it. I guess if the predicted result itself is wrong, or a certain parameter I set is wrong.

[Sekunde]

the ply file is in the world space (z up); whereas the voxels (see my screenshot) are stored in the .scene file in the voxel space (y up). I assume there is already a problem with voxel space.

can you share with me the data that you aligned (especially the .scene file)?

[Maple-3d]

Hi！ I think what you said is correct, because in my test I found that after I changed the coordinate system of the scene, the final output visualization results also changed This is the .scene file create from your datagen script. scene0529_000.zip Can you tell me how to visiualize this file?

[Sekunde]

python tools/visualization.py --path example_result/test/ this will generate the visualisation of voxels as ply files under example_result/test/scene0529_00

from my end, it looks okish now in voxel space, there are probably then some transformation error from voxel space to mesh space.

[Sekunde]

please re-open when further questions