Rendering multiple times results in incresingly higher memory usage and slower processing

[mikkelmedm]

Describe the bug Not sure if this is a bug, or if I can somehow optimize my code more. Sorry if I have missed something obvious.

Yesterday I started a process of running multiple renders (15 frames * 1000 runs), it started out running fine ("Fra: n Mem: ~2.700M"). However inspecting this morning each render took waay longer and memory was "Fra: n Mem: ~12.000M".

Trying to rerun the process now, I notice also that Memory is increasing slighty for each run.

General Information

1. Which BlenderProc version are you using?

Version 2.0.0a7

2. On which operating system are you?

Ubuntu

3. Have you checked the issue tracker to see if a similar issue has been opened?

Yes.

4. Have you changed BlenderProc in any way besides the config file? If yes, are you sure that this change does not affect the problem you are having?

No, just the created my own main.py file.

To Reproduce Steps to reproduce the behavior:

1. Provide the full command you used to run BlenderProc: blenderproc run main.py --scene "beer.blend" --object_name "beer"
2. Provide the full python file, you used:

   
   import argparse
   import os
   import numpy as np
   from numpy.random import choice
   import random
   import bpy
   import importlib
   from mathutils import Vector
   from blenderproc.python.types.MeshObjectUtility import MeshObject
   import time

parser = argparse.ArgumentParser()

Paths:

parser.add_argument('--bop_parent_path', default="bop_dataset", help="Path to the bop datasets parent directory") parser.add_argument('--cc_textures_path', default="resources/cctextures", help="Path to downloaded cc textures") parser.add_argument('--output_dir', default="trackar_data/output", help="Path to where the final files will be saved ")

Parameters:

parser.add_argument('--bound_box', default=False, help="True for BBox keypoints - False for unique keypoints") parser.add_argument('--stand_vs_lay_prob', default=[0.7, 0.3], help="Propability of standing object vs laying object, respectively")

parser.add_argument('--scene', default="trackar_data/chairs.blend", help="Path to the scene.blend file") parser.add_argument('--object_name', default="Can", help="Name of object in .blend file with keypoints") parser.add_argument('--multi_obj', default=False, help="If the scene contains multiple objects to load one at random")

parser.add_argument('--num_runs', default=774, help="Number of runs") parser.add_argument('--frames_per_run', default=15, help="Number of frames for each run") parser.add_argument('--out_size', default=(256, 512), help="Output width and output height") parser.add_argument('--cam_dist_min', default=0.25, help="Minimum radius of camera sampler shell") parser.add_argument('--cam_dist_max', default=0.8, help="Maximum radius of camera sampler shell") parser.add_argument('--distractor_objs_min', default=2, help="Minimum number of occluding/distractor objects to add to the scene") parser.add_argument('--distractor_objs_max', default=13, help="Maximum number of occluding/distractor objects to add to the scene") args = parser.parse_args()

bproc.init()

Set to True for faster outputs during debbuging

DEBUG = False

If DEBUG==True it loads only a random sample of bop objects, otherwise loads all objects ONCE

at start of script and randomly chooses n-size subset for each run to place in scene.

distractor_bop_objs = bproc.loader.load_bop(bop_dataset_path = os.path.join(args.bop_parent_path, 'hb'), sys_paths = "bop_toolkit", sample_objects=True if DEBUG else False, num_of_objs_to_sample=2 if DEBUG else None, mm2m = True) distractor_bop_objs += bproc.loader.load_bop(bop_dataset_path = os.path.join(args.bop_parent_path, 'ycbv'), sys_paths = "bop_toolkit", sample_objects=True if DEBUG else False, num_of_objs_to_sample=2 if DEBUG else None, mm2m = True)

if DEBUG: bproc.renderer.set_samples(10) distractor_objs_max = 2 distractor_objs_min = 1

else: distractor_objs_max = args.distractor_objs_max distractor_objs_min = args.distractor_objs_min bproc.renderer.set_light_bounces(
diffuse_bounces = 200, glossy_bounces = 200, max_bounces = 200, transmission_bounces = 200, transparent_max_bounces = 200 )

set segmap_class (category_id) and physics properties of distractor objects:

for j, obj in enumerate(distractor_bop_objs): obj.set_cp("category_id", 0)

load the blender object(s) into the scene

load_blend_objs = bproc.loader.load_blend(args.scene)

If we need to scale the object:

scale = 0.15

set segmap_class (category_id) and physics properties

for obj in load_blend_objs: obj.set_cp("category_id", 1)

if not args.multi_obj and obj.get_name() == args.object_name:
    obj_of_interest = obj 
    object_name = obj_of_interest.get_name()
    obj.set_scale(Vector((scale, scale, scale)))

Load cctextures:

cc_textures = bproc.loader.load_ccmaterials(args.cc_textures_path)

light_point = bproc.types.Light() light_point.set_energy(200)

Iteratively generate a new scene with random camera/light/object placements:

for r in range(args.num_runs):

# If .blend file contains more objects, select one at random, hide the others:
if args.multi_obj:
    obj_of_interest = np.random.choice(load_blend_objs)
    object_name = obj_of_interest.get_name()
    obj_of_interest.set_scale(Vector((scale, scale, scale)))
    for obj in load_blend_objs:
        obj.hide(True)
    obj_of_interest.hide(False)

bproc.utility.reset_keyframes()
FOV = np.random.uniform(1.04, 1.277) 
bproc.camera.set_intrinsics_from_blender_params(FOV, args.out_size[0], args.out_size[1], lens_unit="FOV")

# Sample a random size subset of distractor objects each run:
distractor_bop_objs_sample = random.sample(distractor_bop_objs, random.randint(distractor_objs_min, distractor_objs_max))

# Hide/unhide the different objects during renders
for obj in distractor_bop_objs:
    obj.hide(True)

for obj in distractor_bop_objs_sample:
    obj.hide(False)

draw = choice([True, False], 1, p=args.stand_vs_lay_prob)
SURFACE = draw.item(0)

# create room
room_planes = [bproc.object.create_primitive('PLANE', scale=[2, 2, 1]),
            bproc.object.create_primitive('PLANE', scale=[2, 2, 1], location=[0, -2, 2], rotation=[-1.570796, 0, 0]),
            bproc.object.create_primitive('PLANE', scale=[2, 2, 1], location=[0, 2, 2], rotation=[1.570796, 0, 0]),
            bproc.object.create_primitive('PLANE', scale=[2, 2, 1], location=[2, 0, 2], rotation=[0, -1.570796, 0]),
            bproc.object.create_primitive('PLANE', scale=[2, 2, 1], location=[-2, 0, 2], rotation=[0, 1.570796, 0])]

if not SURFACE:   
    obj_of_interest.enable_rigidbody(True, friction = 100.0, linear_damping = 0.99, angular_damping = 0.99)

    for plane in room_planes:
        plane.enable_rigidbody(False, collision_shape='BOX', friction = 100.0, linear_damping = 0.99, angular_damping = 0.99)

    for obj in (distractor_bop_objs_sample):
            obj.enable_rigidbody(True, friction = 100.0, linear_damping = 0.99, angular_damping = 0.99)

# sample light color and strength from ceiling
light_plane = bproc.object.create_primitive('PLANE', scale=[3, 3, 1], location=[0, 0, 10])
light_plane.set_name('light_plane')
light_plane_material = bproc.material.create('light_material')
light_plane_material.make_emissive(emission_strength=np.random.uniform(3,6), replace=True,
                                emission_color=np.random.uniform([0.5, 0.5, 0.5, 1.0], [1.0, 1.0, 1.0, 1.0]))    
light_plane.replace_materials(light_plane_material)

# sample point light on shell

light_point.set_color(np.random.uniform([0.5, 0.5, 0.5], [1, 1, 1]))
location = bproc.sampler.shell(center = [0, 0, 0], radius_min = 1, radius_max = 1.5,
                        elevation_min = 5, elevation_max = 89, uniform_elevation = True)
light_point.set_location(location)

# sample CC Texture and assign to room planes
random_cc_texture = np.random.choice(cc_textures)
for idx, plane in enumerate(room_planes):
    plane.replace_materials(random_cc_texture)

if SURFACE: # standing
    # Define a function that samples the initial pose of a given object above the ground 
    def sample_initial_pose(obj: bproc.types.MeshObject):
        obj.set_location(bproc.sampler.upper_region(objects_to_sample_on=room_planes[0:1],
                                                    min_height=1, max_height=4, face_sample_range=[0.4, 0.6]))
        obj.set_rotation_euler(np.random.uniform([0, 0, 0], [0, 0, np.pi * 2]))

    # Sample objects on the given surface
    placed_objects = bproc.object.sample_poses_on_surface(objects_to_sample=[obj_of_interest] + distractor_bop_objs_sample,
                                            surface=room_planes[0],
                                            sample_pose_func=sample_initial_pose,
                                            min_distance=0.01,
                                            max_distance=0.4)

    # BVH tree used for camera obstacle checks
    bvh_tree = bproc.object.create_bvh_tree_multi_objects(placed_objects)

else: # laying
    # Define a function that samples 6-DoF poses
    def sample_pose_func(obj: bproc.types.MeshObject):
        min = np.random.uniform([-0.3, -0.3, 0.0], [-0.2, -0.2, 0.0])
        max = np.random.uniform([0.2, 0.2, 0.4], [0.3, 0.3, 0.6])
        obj.set_location(np.random.uniform(min, max))
        obj.set_rotation_euler(bproc.sampler.uniformSO3())

    # Sample object poses and check collisions 
    bproc.object.sample_poses(objects_to_sample = [obj_of_interest] + distractor_bop_objs_sample,
                            sample_pose_func = sample_pose_func, 
                            max_tries = 1000)

    # Physics Positioning
    bproc.object.simulate_physics_and_fix_final_poses(min_simulation_time=3,
                                                    max_simulation_time=10,
                                                    check_object_interval=1,
                                                    substeps_per_frame = 20,
                                                    solver_iters=25)
    # BVH tree used for camera obstacle checks
    bvh_tree = bproc.object.create_bvh_tree_multi_objects([obj_of_interest] + distractor_bop_objs)

poses = 0
while poses < args.frames_per_run:
    # Sample location
    location = bproc.sampler.shell(center = obj_of_interest.get_location(),
                            radius_min = args.cam_dist_min,
                            radius_max = args.cam_dist_max,
                            elevation_min = 10,
                            elevation_max = 45,
                            uniform_elevation = True)

    # Determine point of interest in scene
    poi = bproc.object.compute_poi([obj_of_interest])

    # Add variation to the poi (so camera is not always statically pointing at obj_of_interest)
    vari = [0,0,0]
    vari[random.randint(0, len(vari)-1)] = random.uniform(0, 0.1)
    poi += vari

    # Compute rotation based on vector going from location towards poi
    rotation_matrix = bproc.camera.rotation_from_forward_vec(poi - location, inplane_rot=np.random.uniform(-0.7854, 0.7854))
    # Add homog cam pose based on location an rotation
    cam2world_matrix = bproc.math.build_transformation_mat(location, rotation_matrix)

    # Check that obstacles are at least 0.2 meter away from the camera and make sure the view interesting enough
    if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.2}, bvh_tree):
        # Persist camera pose
        bproc.camera.add_camera_pose(cam2world_matrix)
        poses += 1

# render the whole pipeline
data = bproc.renderer.render()
# Render segmentation masks (per class and per instance)
data.update(bproc.renderer.render_segmap(map_by=["class"]))
# Write data in custom format -> "blenderproc/python/writer/CustomWriter.py"
bproc.writer.write_custom(os.path.join(args.output_dir), data, object_name, args.bound_box, append_to_existing_output=True)

**Expected behavior**
Suspect the cc_textures to take up the memory over time, since, when inspecting this morning the slow running process, it looked like it was looping over all/alot of the cc_textures each run(?). 
However, expected the process to not take up increasingly more memory and run slower for each run. 

Hope you can help - and thanks again for this awesome project!

[themasterlink]

Hey,

this is a typical problem, one way to solve this is to rerun BlenderProc multiple times to avoid lingering memory demands from past allocations. Right now we do not offer a function to remove orphan unused data.

I would recommend to sub sample the amount of cctextures in the first part of the script to maybe 50 at maximum and then rerun this particular script 10 times. You will loose a bit of time for the initial setup, but it will be much faster overall.

I hope this helps.

Best, Max

[mikkelmedm]

Hi :) Thanks for your quick response!

With the new version, is there an straightforward way of doing the rerun (similar to the rerun.py script in previous version)?

[themasterlink]

Yes, you can just use the rerun.py script. That is the correct way of doing it.

[mikkelmedm]

Sorry for asking, I cannot seem to figure it out. So if I would usually run above main.py script with e.g. ´´blenderproc run main.py --scene "beer.blend"´´, would I then have to now run ´´python rerun.py main.py --scene "beer.blend"´´? Thanks

[themasterlink]

Yes exactly :)

Best, Max

[mikkelmedm]

So I usually run blenderproc run main.py --scene "beer.blend". Now running python rerun.py main.py --scene "beer.blend" leaves me with the following error message:

[mikkelmedm]

Sorry nevermind, I had to specify the output folder as the last argument, when using rerun.py, forgot about that :)