[BUG]: fail in visualization

qsun1 commented 1 year ago

Hi, sorry to bother you again.

After completed in rendering step, I met another bug in visualization in 2D and 3D with

CUDA_VISIBLE_DEVICES=7, \
python visualization/front3d/vis_front3d.py --json_file 01a90e65-5653-4b48-88fa-4aa780db0621.json

Is /usr/bin/xdg-open: 869: www-browser: not found etc. mean that my device does not include the software that can open the images rendered?
another question is about "no instances to display?" Does it mean that some error in the last rendering step? The error comes from
```
viser_2D.draw_colors()
viser_2D.draw_depths()
viser_2D.draw_cls_maps()
viser_2D.draw_inst_maps(type=('mask'))
```
or https://github.com/yinyunie/BlenderProc-3DFront/blob/7db2d0cad41424bba682827f89e179965589d362/visualization/front3d/vis_front3d.py#L119

Actually I can open the 3 tmp pictures /tmp/tmp_35gfew1.PNG /tmp/tmp67is7qq3.PNG /tmp/tmp43va1dpr.PNG that include rgb images, depth images and mask

qsun1 commented 1 year ago

Some tmp results

The first picture is too big to load up, which is just 10x10 RGB image grid. Similarly, the last two pictures is tmp67is7qq3 tmp43va1dpr

qsun1 commented 1 year ago

Another try

I neglected the aforementioned bug by commenting out this line https://github.com/yinyunie/BlenderProc-3DFront/blob/7db2d0cad41424bba682827f89e179965589d362/visualization/front3d/vis_front3d.py#L119

However, it comes up with another bug in this line: https://github.com/yinyunie/BlenderProc-3DFront/blob/7db2d0cad41424bba682827f89e179965589d362/visualization/front3d/vis_front3d.py#L121

I try to print the variable focused_3dboxes and it turns to be [], a empty list. Does it have connection with the last log no instances to display ?

qsun1 commented 1 year ago

additional results

2x2 RGB image grid

The first picture is too big to load up, which is just 10x10 RGB image grid. Similarly, the last two pictures is

qsun1 commented 1 year ago

In https://github.com/yinyunie/BlenderProc-3DFront/blob/7db2d0cad41424bba682827f89e179965589d362/visualization/front3d/vis_front3d.py#L106, the instance_attrs turns to be [[], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], [], []]. That is abnormal, although this is the reason why "no instances " reported in the code. Does it mean that no instances are in the scene in actual fact ?

yinyunie commented 1 year ago

Hi, just as in the README, you need X server to display. That means you may need a monitor to showcase the visualizations because the scripts below will pop up a window to present the visualization.

viser_2D.draw_colors()
viser_2D.draw_depths()
viser_2D.draw_cls_maps()
viser_2D.draw_inst_maps(type=('mask'))
# and
viser.visualize(...)

JonasSchult commented 1 year ago

Hi Yinyu,

thanks for sharing this codebase! :)

I encounter the same issue as Qi Sun (here). All elements in instance_attribute_mapping only contain dicts with inst_mark set to to layout_XXX (XXX is an integer number), which are filtered out here.

Do you have an idea how to fix this?

Thank you so much, Jonas

qsun1 commented 1 year ago

Hi, just as in the README, you need X server to display. That means you may need a monitor to showcase the visualizations because the scripts below will pop up a window to present the visualization.
viser_2D.draw_colors()
viser_2D.draw_depths()
viser_2D.draw_cls_maps()
viser_2D.draw_inst_maps(type=('mask'))
# and
viser.visualize(...)

Thanks for your advice, after setting the display of my remote device, the window poped up like this. But if I do understand correctly, I may encounter the same issue as Jonas that the instances are all filtered out and I have no idea about this...

yinyunie commented 1 year ago

Hi Yinyu,

thanks for sharing this codebase! :)

I encounter the same issue as Qi Sun (here). All elements in instance_attribute_mapping only contain dicts with inst_mark set to to layout_XXX (XXX is an integer number), which are filtered out here.

Do you have an idea how to fix this?

Thank you so much, Jonas

Hi,

Thanks for the update. This is quite strange. If you also use the latest 3D-Front, the only difference could be the Blender version. BlenderProc automatically installed the blender-3.0.0-linux-x64 in my system. Could you please have a try?

The inst_mark in my instance_attribute_mapping contains both layout_XX and furniture_XX like below

[inst['inst_mark'] for inst in instance_attribute_mapping]
['layout_45', 'layout_49', 'layout_50', 'layout_61', 'layout_62', 'layout_70', 'layout_71', 'layout_73', 'layout_79', 'layout_215', 'layout_216', 'layout_234', 'layout_239', 'layout_259', 'layout_260', 'layout_262', 'layout_263', 'layout_264', 'layout_289', 'layout_290', 'layout_292', 'layout_294', 'layout_296', 'layout_297', 'layout_299', 'layout_301', 'layout_303', 'layout_304', 'layout_306', 'layout_308', 'layout_317', 'layout_318', 'layout_322', 'layout_326', 'layout_327', 'layout_333', 'layout_334', 'layout_335', 'layout_336', 'layout_337', 'layout_338', 'layout_340', 'layout_341', 'layout_342', 'layout_344', 'layout_345', 'furniture_35', 'furniture_25', 'furniture_26', 'furniture_26', 'furniture_26', 'furniture_26', 'furniture_33', 'furniture_27', 'furniture_27', 'furniture_27', 'furniture_27', 'furniture_28', 'furniture_28', 'furniture_28', 'furniture_28', 'furniture_29', 'furniture_29', 'furniture_29', 'furniture_29']

From your renderings, I can see furniture objects are successfully loaded, but their inst_mark attributes may not be correctly read or written.

There are two types of objects furniture(indoor objects) and layout(walls, ceilings, floors). Furniture's inst_mark attribute is loaded here through here. Layout's inst_mark (i.e., layout_XX) is loaded via here

You can check the above to see if they are successfully loaded. If yes, then the problem would happen in the export part. Then you may need to check here.

Sorry for the limited information, I cannot locate the exact bug. My gut feeling is that we probably used a different Blender version. It is worthwhile to first try blender-3.0.0-linux-x64 before debugging.

Best, Yinyu

yinyunie commented 1 year ago

I also added the Blender information in README here.

qsun1 commented 1 year ago

Hi Yinyu, thanks for sharing this codebase! :) I encounter the same issue as Qi Sun (here). All elements in instance_attribute_mapping only contain dicts with inst_mark set to to layout_XXX (XXX is an integer number), which are filtered out here. Do you have an idea how to fix this? Thank you so much, Jonas

Hi,

Thanks for the update. This is quite strange. If you also use the latest 3D-Front, the only difference could be the Blender version. BlenderProc automatically installed the blender-3.0.0-linux-x64 in my system. Could you please have a try?

The inst_mark in my instance_attribute_mapping contains both layout_XX and furniture_XX like below
[inst['inst_mark'] for inst in instance_attribute_mapping]
['layout_45', 'layout_49', 'layout_50', 'layout_61', 'layout_62', 'layout_70', 'layout_71', 'layout_73', 'layout_79', 'layout_215', 'layout_216', 'layout_234', 'layout_239', 'layout_259', 'layout_260', 'layout_262', 'layout_263', 'layout_264', 'layout_289', 'layout_290', 'layout_292', 'layout_294', 'layout_296', 'layout_297', 'layout_299', 'layout_301', 'layout_303', 'layout_304', 'layout_306', 'layout_308', 'layout_317', 'layout_318', 'layout_322', 'layout_326', 'layout_327', 'layout_333', 'layout_334', 'layout_335', 'layout_336', 'layout_337', 'layout_338', 'layout_340', 'layout_341', 'layout_342', 'layout_344', 'layout_345', 'furniture_35', 'furniture_25', 'furniture_26', 'furniture_26', 'furniture_26', 'furniture_26', 'furniture_33', 'furniture_27', 'furniture_27', 'furniture_27', 'furniture_27', 'furniture_28', 'furniture_28', 'furniture_28', 'furniture_28', 'furniture_29', 'furniture_29', 'furniture_29', 'furniture_29']
From your renderings, I can see furniture objects are successfully loaded, but their inst_mark attributes may not be correctly read or written.

There are two types of objects furniture(indoor objects) and layout(walls, ceilings, floors). Furniture's inst_mark attribute is loaded here through here. Layout's inst_mark (i.e., layout_XX) is loaded via here

You can check the above to see if they are successfully loaded. If yes, then the problem would happen in the export part. Then you may need to check here.

Sorry for the limited information, I cannot locate the exact bug. My gut feeling is that we probably used a different Blender version. It is worthwhile to first try blender-3.0.0-linux-x64 before debugging.

Best, Yinyu

Hi,

Thank you for your help. I do use blender-3.0.0-linux-x64 system. I think there are something wrong in the rendering step. Here is my script

blenderproc run \
examples/datasets/front_3d_with_improved_mat/render_dataset_improved_mat.py \
examples/datasets/front_3d_with_improved_mat/3D-FRONT \
examples/datasets/front_3d_with_improved_mat/3D-FUTURE-model \
examples/datasets/front_3d_with_improved_mat/3D-FRONT-texture \
6a0e73bc-d0c4-4a38-bfb6-e083ce05ebe9.json \
resources/cctextures/ \
examples/datasets/front_3d_with_improved_mat/renderings

in my cases,

[inst['inst_mark'] for inst in instance_attribute_mapping]
['', 'layout_3', 'layout_31', 'layout_35']

https://github.com/yinyunie/BlenderProc-3DFront/blob/7db2d0cad41424bba682827f89e179965589d362/examples/datasets/front_3d_with_improved_mat/render_dataset_improved_mat.py#L246 I print all values of inst_mark in instance_attribute_maps between written into the hdf5 file, and it turns out to be

Best, Qi

yinyunie commented 1 year ago

Hi,

I fixed the bugs and got it successfully installed on another Ubuntu machine. I think the bug could be from some version conflicts (numpy) within the built-in python packages in blender. Could you please try again and install from the very beginning?

Before installation, please

delete the blender folder from /home/USER_NAME/
pull the latest codebase (make sure there is no blenderproc.egg-info folder in the project directory)

Then install following the updated README.

If there is still a bug, I zipped my Blender. Please download and put to the same folder in /home/USER_NAME/blender.

Hope it helps.

Best, Yinyu

qsun1 commented 1 year ago

Hi,

My problem is solved. Thank you very much!

Best, Qi

yinyunie / BlenderProc-3DFront