[BUG]: "Exception: There are more object colors than there are objects" using HDRI

[mikkelmedm]

Describe the bug Duplicate of #283. This error seem to pop up after a few runs at a certain angle, where it appears camera is facing the sky. Seem the fix of #283 is already implemented in the newer versions, so very curios to what could help in this scenario.

General Information

1. Which BlenderProc version are you using? Newest version (2.2.0)
2. On which operating system are you? Linux
3. Have you checked the issue tracker to see if a similar issue has been opened? Yes.
4. Have you changed BlenderProc in any way besides the config file? If yes, are you sure that this change does not affect the problem you are having? No. To Reproduce Steps to reproduce the behavior:
5. Provide the full command you used to run BlenderProc: blenderproc run main.py output
6. Provide the full python file, you used:

   
   import blenderproc as bproc
   import argparse
   import os
   import numpy as np
   from numpy.random import choice
   import random
   import bpy
   import importlib
   from mathutils import Vector
   from blenderproc.python.types.MeshObjectUtility import MeshObject
   import time

parser = argparse.ArgumentParser()

Paths:

parser.add_argument('output_dir', default="trackar_data/output", help="Path to where the final files will be saved ") parser.add_argument('--cc_textures_path', default="resources/cctextures", help="Path to downloaded cc textures")

Parameters:

parser.add_argument('--bound_box', default=False, help="True for BBox keypoints - False for unique keypoints")

parser.add_argument('--scene', default="heineken.blend", help="Path to the scene.blend file") parser.add_argument('--object_name', default="beer", help="Name of object in .blend file with keypoints") parser.add_argument('--multi_obj', default=False, help="If the scene contains multiple objects to load one at random")

parser.add_argument('haven_path', nargs='?', default="resources/haven_dataset", help="The folder where the hdri folder can be found, to load an world environment")

parser.add_argument('--num_runs', default=10, help="Number of runs") parser.add_argument('--frames_per_run', default=15, help="Number of frames for each run") parser.add_argument('--out_size', default=(256, 512), help="Output width and output height") parser.add_argument('--cam_dist_min', default=0.25, help="Minimum radius of camera sampler shell") parser.add_argument('--cam_dist_max', default=0.8, help="Maximum radius of camera sampler shell") parser.add_argument('--distractor_objs_min', default=2, help="Minimum number of occluding/distractor objects to add to the scene") parser.add_argument('--distractor_objs_max', default=15, help="Maximum number of occluding/distractor objects to add to the scene") args = parser.parse_args()

bproc.init()

output_dir = "/".join(args.output_dir.split("/")[:-1]) if args.output_dir[-1].isdigit() else args.output_dir

Set to True for faster outputs during debbuging

DEBUG = True

if DEBUG: bproc.renderer.set_max_amount_of_samples(10)

else: distractor_objs_max = args.distractor_objs_max distractor_objs_min = args.distractor_objs_min bproc.renderer.set_light_bounces(
diffuse_bounces = 200, glossy_bounces = 200, max_bounces = 200, transmission_bounces = 200, transparent_max_bounces = 200 ) bproc.renderer.set_max_amount_of_samples(256) # TODO: Should we leave this as default on 1024 ?? takes longer

load the blender object(s) into the scene

load_blend_objs = bproc.loader.load_blend(args.scene)

scale = 0.15

set segmap_class (category_id) and physics properties

for obj in load_blend_objs: obj.set_cp("category_id", 1)

if not args.multi_obj and obj.get_name() == args.object_name:
    obj_of_interest = obj 
    object_name = obj_of_interest.get_name()
    obj.set_scale(Vector((scale, scale, scale)))

Load subset of cctextures each rerun:

cc_textures = random.sample(bproc.loader.load_ccmaterials(args.cc_textures_path), 50)

light_point = bproc.types.Light() light_point.set_energy(200)

Iteratively generate a new scene with random camera/light/object placements:

for r in range(args.num_runs):

# If .blend file contains more objects, select one at random, hide the others:
if args.multi_obj:
    obj_of_interest = np.random.choice(load_blend_objs)
    object_name = obj_of_interest.get_name()
    obj_of_interest.set_scale(Vector((scale, scale, scale)))
    for obj in load_blend_objs:
        obj.hide(True)
    obj_of_interest.hide(False)

bproc.utility.reset_keyframes()
FOV = np.random.uniform(1.04, 1.277) 
bproc.camera.set_intrinsics_from_blender_params(FOV, args.out_size[0], args.out_size[1], lens_unit="FOV")

# Collect all materials
materials = bproc.material.collect_all()

# Set a random hdri from the given haven directory as background
haven_hdri_path = bproc.loader.get_random_world_background_hdr_img_path_from_haven(args.haven_path)
bproc.world.set_world_background_hdr_img(haven_hdri_path)

room_planes = [bproc.object.create_primitive('PLANE', scale=[2, 2, 1])]

# sample light color and strength from ceiling
light_plane = bproc.object.create_primitive('PLANE', scale=[3, 3, 1], location=[0, 0, 10])
light_plane.set_name('light_plane')
light_plane_material = bproc.material.create('light_material')
light_plane_material.make_emissive(emission_strength=np.random.uniform(3,6), replace=True,
                                emission_color=np.random.uniform([0.5, 0.5, 0.5, 1.0], [1.0, 1.0, 1.0, 1.0]))    
light_plane.replace_materials(light_plane_material)

# sample point light on shell

light_point.set_color(np.random.uniform([0.5, 0.5, 0.5], [1, 1, 1]))
location = bproc.sampler.shell(center = [0, 0, 0], radius_min = 1, radius_max = 1.5,
                        elevation_min = 5, elevation_max = 89)
light_point.set_location(location)

# sample CC Texture and assign to room planes
random_cc_texture = np.random.choice(cc_textures)
for idx, plane in enumerate(room_planes):
    plane.replace_materials(random_cc_texture)

# Define a function that samples the initial pose of a given object above the ground 
def sample_initial_pose(obj: bproc.types.MeshObject):
    obj.set_location(bproc.sampler.upper_region(objects_to_sample_on=room_planes[0:1],
                                                min_height=1, max_height=4, face_sample_range=[0.4, 0.6]))
    obj.set_rotation_euler(np.random.uniform([0, 0, 0], [0, 0, np.pi * 2]))

# Sample objects on the given surface
placed_objects = bproc.object.sample_poses_on_surface(objects_to_sample=[obj_of_interest],
                                            surface=room_planes[0],
                                            sample_pose_func=sample_initial_pose,
                                            min_distance=0.01,
                                            max_distance=1)

# BVH tree used for camera obstacle checks
bvh_tree = bproc.object.create_bvh_tree_multi_objects(placed_objects)

poses = 0
while poses < args.frames_per_run:
    # Sample location
    location = bproc.sampler.shell(center = obj_of_interest.get_location(),
                            radius_min = args.cam_dist_min,
                            radius_max = args.cam_dist_max,
                            elevation_min = 10,
                            elevation_max = 45)

    # Determine point of interest in scene
    poi = bproc.object.compute_poi([obj_of_interest])

    # Add variation to the poi (so camera is not always statically pointing at obj_of_interest)
    vari = [0,0,0]
    vari[random.randint(0, len(vari)-1)] = random.uniform(0, 0.1)
    poi += vari

    # Compute rotation based on vector going from location towards poi
    rotation_matrix = bproc.camera.rotation_from_forward_vec(poi - location, inplane_rot=np.random.uniform(-0.7854, 0.7854))
    # Add homog cam pose based on location an rotation
    cam2world_matrix = bproc.math.build_transformation_mat(location, rotation_matrix)

    # Check that obstacles are at least 0.2 meter away from the camera and make sure the view interesting enough
    if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.2}, bvh_tree):
        # Persist camera pose
        bproc.camera.add_camera_pose(cam2world_matrix)
        poses += 1

# render the whole pipeline
data = bproc.renderer.render()
# Render segmentation masks (per class and per instance)
data.update(bproc.renderer.render_segmap(map_by=["class"]))
# Write data in custom format -> "blenderproc/python/writer/CustomWriter.py"
bproc.writer.write_custom(os.path.join(output_dir), data, object_name, args.bound_box, append_to_existing_output=True)

3. Provide a link to all 3D models you used, if they are from one of the publicly available supported datasets, provide the name or path so that it is possible to reproduce the error.
https://drive.google.com/file/d/1piSHdvrguewaVdPGTkY5O_8rrdmlpByq/view?usp=sharing

[mikkelmedm]

Okay, so guess I found a workable solution. But not sure how good it is. In utility/SegMapRendererUtility.py I changed it to

def _set_world_background_color(color):
        """ Set the background color of the blender world obejct.

        :param color: A 3-dim array containing the background color in range [0, 255]
        """
        nodes = bpy.context.scene.world.node_tree.nodes
        links = bpy.context.scene.world.node_tree.links

        # Unlink any incoming link 
        if len(nodes.get("Background").inputs['Color'].links) > 0:
            links.remove(nodes.get("Background").inputs['Color'].links[0])
        # Set strength to 1 as it would act as a multiplier
        nodes.get("Background").inputs['Strength'].default_value = 0 # CHANGED THIS TO 0 INSTEAD OF 1
        nodes.get("Background").inputs['Color'].default_value = color + [0] # CHANGED THIS TO 0 INSTEAD OF 1

This seem to avoid the error, but not knowledable enough in Blender to know the consequences.

[mikkelmedm]

Above didn't help. Still working on a fix.

[mikkelmedm]

Still looking for a fix to this

[saprrow]

Same issue, hope for solution！ T^T。

[cornerfarmer]

Hey @mikkelmedm and @saprrow,

sorry for the late response. Apparently the hdr image was still active during rendering the seg maps which lead to having more colors than objects in the image. You can fix the error by adding two lines to the end of _set_world_background_color:

def _set_world_background_color(color: mathutils.Vector):
    """ Set the background color of the blender world object.

    :param color: A 3-dim array containing the background color in range [0, 255]
    """
    nodes = bpy.context.scene.world.node_tree.nodes
    links = bpy.context.scene.world.node_tree.links

    # Unlink any incoming link that would overwrite the default value
    if len(nodes.get("Background").inputs['Color'].links) > 0:
        links.remove(nodes.get("Background").inputs['Color'].links[0])
    # Set strength to 1 as it would act as a multiplier
    nodes.get("Background").inputs['Strength'].default_value = 1
    nodes.get("Background").inputs['Color'].default_value = color + [1]

    # Make sure the background node is connected to the output node
    output_node = Utility.get_the_one_node_with_type(nodes, "Output")
    links.new(nodes.get("Background").outputs["Background"], output_node.inputs["Surface"])

I will create a PR to fix this on the main branch