Segmentation mask of shadow

[Ben-Nupa]

Hello,

I'm working on a computer vision project in which I try to use synthetic data for image segmentation. Your library can already generate a segmentation map of the objects in a scene. However, I also need to have the mask of the shadow projected by the objects. Is it possible to do that with bpycv ?

For example, left image is the rendered image by Blender (with Render / Render Image) and right image is what I want. Currently, when I generate the segmentation mask with bpycv, I only have the red part (wind turbines) but I would like to add the green part (shadow). Note that the shadow is not rendered by bpycv when I save the result["image"] array (just like in the given examples).

Thanks

[DIYer22]

Sorry! bpycv can't do that. It's not easy to implement shadow's mask.

[Ben-Nupa]

Alright, thanks for your answer, I managed to do what I wanted but it's not very efficient as it requires Blender to render the image twice. If you're interested, here's the code below: I first render the image with bpycv to get the segmentation mask of the objects without the shadow, then I re-render the image with bpy and saves it with RGBA channels, then takes the difference to only get the shadow.

I think it could be done with only one render but to do so, I didn't exactly understand how you generate the segmentation mask. Could you explain it a little ?

import numpy as np
import cv2
import bpy
import bpycv

# Settings
bpy.context.scene.render.image_settings.color_mode = 'RGBA'

# Render image without shadow
result = bpycv.render_data(render_image=False)

# Get segmentation masks of objects
segmentation_map = np.zeros((*result['inst'].shape, 3), dtype=np.uint8)
wind_turbines_mask = result['inst'] == -1
segmentation_map[wind_turbines_mask, 2] = 255  # Objects are colored in red

# Render with image with shadow
bpy.ops.render.render(write_still=True)

# Take difference
rgba = cv2.imread(bpy.context.scene.render.filepath, cv2.IMREAD_UNCHANGED)
shadow_mask = (rgba[..., -1] > 2**(int(bpy.context.scene.render.image_settings.color_depth) -1)) & ~wind_turbines_mask
segmentation_map[shadow_mask, 1] = 255  # Shadow in green
cv2.imwrite('<a path>', segmentation_map)

[DIYer22]

like this:

1. clear all objects's material
2. add Emission Node to each object, and set a unique color according to object's inst_id.
3. render image, and decode the color to inst_id

It's seems not support to render shadow in this pipline.

[Ben-Nupa]

Ok, thank you for your answers. I close the issue then.