How to render the optical flow information of the background？

[jingziyou]

Describe the issue

key step1: load background image from haven_hdri, code as follows： haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(path_dict['haven']) haven_hdri_path = osp.join(assert_root,"HDRI_haven/hdris/altanka/altanka_2k.hdr") print(f'Scene background file path is : {haven_hdri_path}') bproc.world.set_world_background_hdr_img(haven_hdri_path, strength=0.3, rotation_euler=(0,0,0)) key step 2: render flow info, code as follows data = bproc.renderer.render()
data.update(bproc.renderer.render_optical_flow(get_backward_flow=True, get_forward_flow=True, blender_image_coordinate_style=False))

The optical flow information in the background region is 0，What went wrong and what Settings should be added to render background optical flow？

Minimal code example

haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(path_dict['haven'])
  haven_hdri_path = osp.join(assert_root,"HDRI_haven/hdris/altanka/altanka_2k.hdr")
  print(f'Scene background file path is : {haven_hdri_path}')
  bproc.world.set_world_background_hdr_img(haven_hdri_path, strength=0.3, rotation_euler=(0,0,0)) 

  data = bproc.renderer.render()  
  data.update(bproc.renderer.render_optical_flow(get_backward_flow=True, get_forward_flow=True, blender_image_coordinate_style=False))

Files required to run the code

No response

Expected behavior

The optical flow information in the background region is 0，What went wrong and what Settings should be added to render background optical flow？

BlenderProc version

v2.5.1

[cornerfarmer]

Hey @jingziyou,

blender does not support rendering the optical flow of the background. One thing you could do is calculate it by yourself: The background is basically on a sphere with infinite radius, so it should be possible to calculate the optical flow based on the camera poses. Alternatively, you could try to add the hdri to an actual sphere object surrounding the object (see e.g. https://blender.stackexchange.com/questions/207023/how-add-hdri-image-on-sphere). In that way it should occur in the optical flow.