DLR-RM / BlenderProc

A procedural Blender pipeline for photorealistic training image generation
GNU General Public License v3.0
2.8k stars 451 forks source link

How to render the optical flow information of the background? #1146

Open jingziyou opened 2 months ago

jingziyou commented 2 months ago

Describe the issue

key step1: load background image from haven_hdri, code as follows: haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(path_dict['haven']) haven_hdri_path = osp.join(assert_root,"HDRI_haven/hdris/altanka/altanka_2k.hdr") print(f'Scene background file path is : {haven_hdri_path}') bproc.world.set_world_background_hdr_img(haven_hdri_path, strength=0.3, rotation_euler=(0,0,0)) key step 2: render flow info, code as follows data = bproc.renderer.render()
data.update(bproc.renderer.render_optical_flow(get_backward_flow=True, get_forward_flow=True, blender_image_coordinate_style=False))

The optical flow information in the background region is 0,What went wrong and what Settings should be added to render background optical flow? 0_colors_animation 0_forward_flow_animation

Minimal code example

haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(path_dict['haven'])
  haven_hdri_path = osp.join(assert_root,"HDRI_haven/hdris/altanka/altanka_2k.hdr")
  print(f'Scene background file path is : {haven_hdri_path}')
  bproc.world.set_world_background_hdr_img(haven_hdri_path, strength=0.3, rotation_euler=(0,0,0)) 

  data = bproc.renderer.render()  
  data.update(bproc.renderer.render_optical_flow(get_backward_flow=True, get_forward_flow=True, blender_image_coordinate_style=False))

Files required to run the code

No response

Expected behavior

The optical flow information in the background region is 0,What went wrong and what Settings should be added to render background optical flow?

BlenderProc version

v2.5.1

cornerfarmer commented 2 months ago

Hey @jingziyou,

blender does not support rendering the optical flow of the background. One thing you could do is calculate it by yourself: The background is basically on a sphere with infinite radius, so it should be possible to calculate the optical flow based on the camera poses. Alternatively, you could try to add the hdri to an actual sphere object surrounding the object (see e.g. https://blender.stackexchange.com/questions/207023/how-add-hdri-image-on-sphere). In that way it should occur in the optical flow.