[Question]: Physics are not rendered the same way as in Blender / objects do not fall fast enough

[danbenos]

Describe the bug Certain objects do not emulate physics like they would if seen in Blender animation, or they take much longer to fall down.

General Information

1. Which BlenderProc version are you using? 2.2.0

2. On which operating system are you? Ubuntu 20.04.2 LTS

3. Have you checked the issue tracker to see if a similar issue has been opened? Yes, I saw #251, but my problem should be much simpler, and I cannot find where I can untick Simplify at the bottom of that issue ticket.

4. Have you changed BlenderProc in any way besides the config file? If yes, are you sure that this change does not affect the problem you are having? No

To Reproduce Steps to reproduce the behavior:

1. Provide the full command you used to run BlenderProc: blenderproc run pipelineLabel.py
2. Provide the full python file, you used (this is a simplified version, with most randomization removed to better visualize the problem):

   
   import blenderproc as bproc
   import argparse
   import numpy as np
   import random
   import os
   import glob

parser = argparse.ArgumentParser() parser.add_argument('output_dir', nargs='?', default="output", help="Path to where the final files, will be saved") args = parser.parse_args()

bproc.init()

Create a ground plane

ground = bproc.object.create_primitive('PLANE', scale=[0.35, 0.35, 1]) ground.enable_rigidbody(False, collision_shape='BOX')

classPrefixes = ["Label"]

modelDirectory = 'model' allObjects = []

for annotationId, classPrefix in enumerate(classPrefixes): list_of_object_paths = glob.glob(f'{modelDirectory}/{classPrefix}*.blend') if not list_of_object_paths: continue classObjects = [] for i in range(2): classObjects.extend(bproc.loader.load_blend(np.random.choice(list_of_object_paths), obj_types=["mesh"])) allObjects.extend(classObjects)

for obj in allObjects: obj.enable_rigidbody(active=True)

allObjects[0].set_rotation_euler([0, np.pi / 2, 0]) allObjects[1].set_rotation_euler([np.pi / 2, 0, 0])

def sample_pose(obj: bproc.types.MeshObject): obj.set_location(bproc.sampler.upper_region( objects_to_sample_on=[ground], min_height=0.1, max_height=0.4 ))

Sample the poses of all objects, while making sure that no objects collide with each other.

allObjects = bproc.object.sample_poses_on_surface( allObjects, ground, sample_pose_func=sample_pose, min_distance=0, max_distance=1.5, max_tries=10 )

define a light and set its location and energy level

light = bproc.types.Light() light.set_type("POINT") light.set_location(np.random.uniform([-2, -2, 3], [2, 2, 5])) light.set_energy(np.random.uniform(500, 1000))

bproc.object.simulate_physics_and_fix_final_poses(min_simulation_time=2, max_simulation_time=100, check_object_interval=1) # comment this out when simulating in Blender

bproc.camera.set_resolution(576, 544) cam2world_matrix = bproc.math.build_transformation_mat([0, 0, 0.7], [0, 0, 0]) bproc.camera.add_camera_pose(cam2world_matrix)

data = bproc.renderer.render() seg_data = bproc.renderer.render_segmap(map_by=["instance", "class"])

bproc.writer.write_coco_annotations(args.output_dir, instance_segmaps=seg_data["instance_segmaps"], instance_attribute_maps=seg_data["instance_attribute_maps"], append_to_existing_output=False, colors=data["colors"], color_file_format="PNG")

3. Provide a link to all 3D models you used, if they are from one of the publicly available supported datasets, provide the name or path so that it is possible to reproduce the error.
https://drive.google.com/file/d/19evp8EToDm17Yw4L3S17-3K1CqMlaveg/view?usp=sharing

**Expected behavior**
Objects are animated as they looked like in Blender, preferably faster, too.

**Screenshots**
How the object look like in BlenderProc from above (according my use case)
![000000](https://user-images.githubusercontent.com/43173272/162382279-e00e3196-39cd-405c-afcf-b9571f2c9d57.png)
From side (for convenience, script modified in the cam2world_matrix)
![000000](https://user-images.githubusercontent.com/43173272/162382398-914ceebd-7522-4064-9347-9fd522bebce7.png)
How it looked like in Blender
![gravitygif](https://user-images.githubusercontent.com/43173272/162382626-f7bd2b32-732f-4473-b5b5-7ce22e8020f1.gif)

**Additional context**
Following the previous ticket, I know position sampling is not an issue because this still happens no matter how high the object is placed above the plane. I have several objects that must fall to a plane, and my camera will be positioned pointing directly down to them. Most objects are simulated properly, but I have several paper-like objects that seems to not fall on the plane properly, as they ended up standing. Even after increasing the simulation time to 100 seconds and increasing substeps_per_frame to 100 (I don't really understand what this parameter do, but I assume they increase the speed of the physics), they still stand up, while I expect they will fall down due to physics.

Since BlenderProc does not visualize in-between frames of the physics simulation, I attempted to look at it directly on Blender, if I run the script above, but with bproc.object.simulate_physics_and_fix_final_poses commented out (as this will freeze every object afterwards), end then run the animation in Blender, I can see as in the gif that they do fall over after 600 frames (25 seconds if it's 24 fps as described in command line output). 

Now I am wondering if there is a setting that prevents a correct simulation being played, and if I can do something about it (and preferably make it faster so I don't have to wait 25 seconds for each run)

[themasterlink]

Hey,

Even after increasing the simulation time to 100 seconds and increasing substeps_per_frame to 100 (I don't really understand what this parameter do, but I assume they increase the speed of the physics), they still stand up, while I expect they will fall down due to physics.

The substeps per frame tell you how many physics simulation steps are performed per frame, a higher number ensures that object which move at a high speed do not go through other objects. Basically improving the precision of the simulation.

Now I am wondering if there is a setting that prevents a correct simulation being played, and if I can do something about it (and preferably make it faster so I don't have to wait 25 seconds for each run)

You don't have to run it inside of blender I assume your problem is the check_object_interval. This checks every X seconds if the objects stopped moving, when they hit plane, they will appear to stop moving and then start falling over after as you said 25 seconds. Apparently, BlenderProc assumes the objects have stopped moving and stops the simulation to not waste time.

The max_time value is only used if all objects are still moving.

So, I would change this check_object_interval for these particular objects, to a much higher value or to the max value and your fine.

Best, Max

[danbenos]

Thanks for explaining the parameters. So I decided to increase it to extreme numbers (interval 200, max 2000, and later 10x that). Unforntunately, the papers are still not falling flat. In fact, for the second one when the interval is 2000 (min 6000, max 20000), BlenderProc stated that the objects have stopped moving after 6000 seconds.

If I reactivate rigid body in blender and resume where it left off, the papers do continue moving and fall.

Do you have an idea what else I can try?

[themasterlink]

Why do they have to fall at all, can't you just randomly rotate them? So that they are not falling on one of the short sides?

[themasterlink]

Just to clarify, we do exactly the same as you do, we just press the "bake" button:

https://github.com/DLR-RM/BlenderProc/blob/53a72ad84e6974a6941556cf26b13989d0c845bb/blenderproc/python/object/PhysicsSimulation.py#L169

I would just add a random rotation to your sample_pose function

[danbenos]

I need to explain the actual context for this. In the original code, the objects are indeed randomly rotated from SO3. However, as it is random, it is not impossible that they still fall on the short side. Sometimes, I am lucky and it falls (almost) flat.

Other times, I am not and this happens:

Therefore, I tried investigating what causes this to happen and hopefully mitigate this from happening. The weird thing here is that other non-paper-shaped objects like the bottle you see are practically never affected from this.

I can write the code and only randomize on Z axis so it stays flat, but I do need other objects to be randomized on all axis.

Original code, to illustrate:

import blenderproc as bproc
import argparse
import numpy as np
import random
import os
import glob

parser = argparse.ArgumentParser()
parser.add_argument('--output_dir', nargs='?', default="output", help="Path to where the final files, will be saved")
args = parser.parse_args()

bproc.init()

# Create a ground plane
ground = bproc.object.create_primitive('PLANE', scale=[0.75, 0.75, 1])
ground.enable_rigidbody(False, collision_shape='BOX')

groundTextures = ["grass", "asphalt"]
materials = bproc.loader.load_ccmaterials("resources/cctextures", groundTextures, preload=True)
random_cc_texture = np.random.choice(materials)
ground.replace_materials(random_cc_texture)
bproc.loader.load_ccmaterials("resources/cctextures", groundTextures, fill_used_empty_materials=True)
ground.enable_rigidbody(active=False, collision_shape="BOX")

classPrefixes = ["Negative", "Cigarette", "Snippet"]

modelDirectory = 'resources/model'
allObjects = []

for annotationId, classPrefix in enumerate(classPrefixes):
    list_of_object_paths = glob.glob(f'{modelDirectory}/*/{classPrefix}*.blend')
    if not list_of_object_paths:
        continue
    classObjects = []
    for i in range(10):
        classObjects.extend(bproc.loader.load_blend(np.random.choice(list_of_object_paths), obj_types=["mesh"]))
    for objects in classObjects:
        objects.set_cp("category_id", annotationId)
    allObjects.extend(classObjects)

# if you move this before posing, can labels now stop standing up?
for obj in allObjects:
    obj.enable_rigidbody(active=True)

def sample_pose(obj: bproc.types.MeshObject):
    obj.set_location(bproc.sampler.upper_region(
        objects_to_sample_on=[ground],
        min_height=0.1,
        max_height=0.4
    ))
    obj.set_rotation_euler(bproc.sampler.uniformSO3())

# Sample the poses of all objects, while making sure that no objects collide with each other.
allObjects = bproc.object.sample_poses_on_surface(
    allObjects,
    ground,
    sample_pose_func=sample_pose,
    min_distance=0.1,
    max_distance=1.5,
    max_tries=10
)

light = bproc.types.Light()
light.set_type("POINT")
light.set_location(np.random.uniform([-2, -2, 3], [2, 2, 5]))
light.set_energy(np.random.uniform(500, 1000))

bproc.object.simulate_physics_and_fix_final_poses(min_simulation_time=20, max_simulation_time=50, check_object_interval=10)

bproc.camera.set_resolution(576, 544)
poses = 0
tries = 0
while tries < 1000 and poses < 3:
    location = np.random.uniform([-0.5, -0.5, 0.5], [0.5, 0.5, 0.5])
    euler_rotation = np.random.uniform([0, 0, 0], [0, 0, np.pi * 2])
    cam2world_matrix = bproc.math.build_transformation_mat(location, euler_rotation)
    if bproc.camera.scene_coverage_score(cam2world_matrix) > 0.1:  # this is the one that calculates if scene is interesting
        # Persist camera pose
        bproc.camera.add_camera_pose(cam2world_matrix)
        poses += 1
    else:
        print("Scene from attempt number " + str(tries) + " to put camera is not interesting enough")
    tries += 1

data = bproc.renderer.render()
seg_data = bproc.renderer.render_segmap(map_by=["instance", "class"])
# don't forget to make it concise and short

bproc.writer.write_coco_annotations(args.output_dir,  # what happens if not joined with coco data? try with rerun.py
                                    instance_segmaps=seg_data["instance_segmaps"],
                                    instance_attribute_maps=seg_data["instance_attribute_maps"],
                                    append_to_existing_output=False,
                                    colors=data["colors"],
                                    color_file_format="PNG")

[themasterlink]

Hey,

the problem is your objects are not truly flat are they? So, they shouldn't even fall over. Why it randomly works in the blender viewer could simply be a different set of default values or one config value, which is switched between what you are using and what is done in the python API.

Lastly, have you tried to give them a thickness? Working with objects without volumes is quite difficult for any physics simulation.

Check out the solidify modifier.

[danbenos]

So I added some thickness this way (see the code visible at the top):

It doesn't fall satisfactorily enough yet. Additionally in this non-random case I can add thickness with confidence, but when loading randomly I am not sure if I can find objects without volumes and and add thickness only to them.

Is this the last possible fix I can try to use?

[themasterlink]

The problem is these paper pieces would not fall in the real world either. Imagine a piece of metal shaped like your paper, if you put in on side, it will just stand there. In the real world it might lean in one direction too much and then fall over, but in the simulation we can't hope for that level of precision of a quick physics simulation.

PS: One last tip for generating data, don't try to make it perfect try to make it as complete as possible. If that means some of the situation examples don't occur in the real world even better, that way you maybe cover a corner case, which will be else super hard to learn. So more randomness in your data is a good thing, removing it might actually hurt your network.

Embrace the chaos is the pro tip for synthetic data generation.

[danbenos]

I understand. I will talk to my colleague and see if I can convince him to agree with that going forward. I will close the ticket for now.

[danbenos]

Actually, I just find something weirder. So I tried loading the papers and do not apply any rotation, the papers are by default laying flat. If I do not activate physics, they will (obviously) stay flat on the plane.

This also applies if I randomize the rotation to SO3, it will stay with whatever rotational pose assigned to it. Now if I simulate physics with BlenderProc, the papers actually stood up from its laying position. See the 0 rotational value on top of the screenshot which is the same as the first one. Running this several times, I see that it actually even favors standing up in its short side (portrait) rather than long side (landscape)

FYI running Blender physics directly from the UI does make them fall if slowly so like gifs above.

I agree metal papers can stand upright, but not from a laying down position. Is this some peculiarity of the mesh I used? Are you certain BlenderProc isn't bugged (not Blender itself since it does simulate correctly in the UI, assuming they are the same), at least against these kinds of mesh?

[themasterlink]

Are you certain BlenderProc isn't bugged (not Blender itself since it does simulate correctly in the UI, assuming they are the same), at least against these kinds of mesh?

No, I am not sure. But, you can check the code yourself. I have no clue why this happens and sadly I have no time to debug your example at the moment.

[themasterlink]

I will close this because of inactivity.

[HannahHaensen]

Hey :) I have a similar issue where my fork is not falling correctly, its always up in the final rendered images, and not laying on the table.

@danbenos where you able to solve your problem?

[danbenos]

Hello Hannah. Unfortunately no, I did not have time to check the code myself, either. Since only a small subset of my 3d object is affected, I decided it's easier to just look for more models that do not stand up.