Segmentation taking more and more time as my script is running

[LudvigDubois]

Hello,

I'm currently working on a python script to produce a car dataset using Blender and I use the zpy library quite a lot (very nice work btw !) .

My script is working like this. I start from a Blender scene containing only a plane (as a shadow catcher) and a camera. Then, I load a random car model and a random HDRI background from some folders. The car model is made of 100+ objects and I segment every object. Then, I move the camera around the car and I make a few renders. After that, I remove the car model and background image and start over with another car and background:

def main(num_scenes = 20):

    saver = zpy.saver_image.ImageSaver(output_dir=OUTPUT_DIR, description='Car dataset')

    hdri_path = Path(Path(bpy.data.filepath).parent, 'hdris')
    hdri_paths = [Path(hdri_path, path) for path in os.listdir(hdri_path)]

    car_path = Path(Path(bpy.data.filepath).parent, 'car_models_processed')
    car_paths = [Path(car_path, path) for path in os.listdir(car_path)]

    image_id = 0

    for scene_id in range(num_scenes):

        # fix camera focal lenght
        bpy.data.cameras[0].lens = 30         

        # load a random background
        zpy.hdris.load_hdri(random.choice(hdri_paths))

        # load a random car and get his size first
        load_car(str(random.choice(car_paths)))

    # parts segmentation
        obj_names = []
        obj_colors = []
        for obj in bpy.data.collections['Car'].all_objects:
            color = (random.random(), random.random(), random.random())
            zpy.objects.segment(obj, name=obj.name, color=color)
            saver.add_category(name=obj.name, color=color)
            obj_names.append(obj.name)
            obj_colors.append(color)

        # move camera around the car
        cam_positions = get_camera_positions()
        for pos in cam_positions:
            target = bpy.data.objects['target']
            camera = bpy.data.objects['Camera']  
            camera.location = pos
            look_at(camera, target)

            rgb_image_name = '%06d' % image_id        
            iseg_image_name = '%06d_iseg' % image_id
            image_id += 1

            zpy.render.render(
                rgb_path=PurePath(OUTPUT_DIR, rgb_image_name + '.png'),
                iseg_path=PurePath(OUTPUT_DIR, iseg_image_name + '.png')
                )

            saver.add_image(
                    name=rgb_image_name + '.png',
                    style='default',
                    output_path=PurePath(OUTPUT_DIR, rgb_image_name + '.png'),
                    frame=image_id
                    )

            saver.add_image(
                    name=iseg_image_name + '.png',
                    style='default',
                    output_path=PurePath(OUTPUT_DIR, iseg_image_name + '.png'),
                    frame=image_id
                    )

            for i in range(len(obj_names)):
                     saver.add_annotation(
                             image=rgb_image_name + '.png',
                             seg_image=iseg_image_name + '.png',
                             seg_color=obj_colors[i],
                             category=obj_names[i]
                             )

            saver.output_annotated_images()
            saver.output_meta_analysis()

            zpy.output_coco.OutputCOCO(saver).output_annotations()
            zpy.output_zumo.OutputZUMO(saver).output_annotations()

        # remove the car and background image
        remove_car()
        remove_background()

The script is working fine, but each time I load a new car, the segmentation part takes 1.5 to 2 seconds more. After a few hundreds cars, the segmentation takes several hundreds of seconds to complete, which makes the script unusable.

After searching for a bit, I found that even if you unlink data blocks from the scene (my functions remove_car() and remove_background() do that), the data blocks still remain in the blend file until you reload it. This is probably what makes the segmentation take more time, but I'm not sure.

In order to fix this, I decided to load a fresh scene after my renderings instead of removing all my data each time. I then replaced the last 2 lines of my code with this line:

bpy.ops.wm.read_homefile(filepath = '/path/to/blend/scene/scene.blend')

Now, the problem is that when this line is called, it seems like I lose some "context" information because I get this error:

Error: Python: Traceback (most recent call last):
  File "C:\Users\Louis\Desktop\Maitrise\test_blender\scene.blend\render.py", line 675, in <module>
  File "C:\Users\Louis\Desktop\Maitrise\test_blender\scene.blend\render.py", line 538, in main
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\config.py", line 1078, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\config.py", line 1055, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\zpy\hdris.py", line 32, in load_hdri
    scene = zpy.blender.verify_blender_scene()
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\config.py", line 1078, in gin_wrapper
    utils.augment_exception_message_and_reraise(e, err_str)
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\utils.py", line 49, in augment_exception_message_and_reraise
    six.raise_from(proxy.with_traceback(exception.__traceback__), None)
  File "<string>", line 3, in raise_from
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\gin\config.py", line 1055, in gin_wrapper
    return fn(*new_args, **new_kwargs)
  File "C:\Program Files\Blender Foundation\Blender 2.92\2.92\python\lib\site-packages\zpy\blender.py", line 165, in verify_blender_scene
    bpy.context.window.scene = scene
AttributeError: 'NoneType' object has no attribute 'scene'
  In call to configurable 'verify_blender_scene' (<function verify_blender_scene at 0x000001FFDC6F3CA8>)
  In call to configurable 'load_hdri' (<function load_hdri at 0x000001FFDC6FB5E8>)

Now, I'm a bit at a loss here, since the Blender documentation on context is quite weak. Does anyone in the Zumo Labs team ever faced this issue? Sorry if this isn't the right place to ask this question.

Thanks a lot!

[hu-po]

Thanks for the detailed question! Re-loading the scene many times within a single run script execution is not a pattern we use or see often, so it is hard to know exactly what is causing the issue (context bugs are notoriously opaque in Blender).

One suggestion would be to change the script so that each execution only loads one car, and then changes the HDRI and camera angle potentially hundreds of times. Camera angle changes and HDRI changes are much cheaper (memory/compute) than loading and segmenting a complicated object like a car. This does mean you would have to run your sim many times to get a variety of cars though (at which point I would recommend our scale backend to make that more convenient).

Appologies for a bit of a non-answer there, but hopefully its a bit helpful. Let us know how progress is going!

[LudvigDubois]

Thanks Hugo for the fast response! I will continue to try figuring out the context bug, but like you said, I think the way to go is to minimize the number of times I load a new car.