[QUESTION]Some questions on render LM format Datasets

[bjutlss]

Recently, I plan to render my custom datasets in the LM format, using examples/datasets/bop_object_on_surface_sampling/main.py.

I have some questions:

1. How do I limit the depth(z in cam_t_m2c) range of the rendered image?

   location = bproc.sampler.shell(center = [0, 0, 0],
                               radius_min = 0.3,
                               radius_max = 1.0,
                               elevation_min = 5,
                               elevation_max = 89)

   I tried to modify the radius_min and radius_max, which seem to live up to expectations, but the depth(z) is sometimes higher or lower. How do I limit the depth range?

2. In this script, the target object will be anchored in the center of the picture. How to make the target object appear randomly in various places in the RGB images? After calculating the poi, I add a random item to adjust the position of object in images,as follows:

   poi = bproc.object.compute_poi(np.random.choice(sampled_target_bop_objs, size=10, replace=True))
   random_sample = np.random.uniform([-0.5, -0.5, 0.0], [0.5, 0.5, 0.0])
   rotation_matrix = bproc.camera.rotation_from_forward_vec(poi + random_sample - location, inplane_rot=np.random.uniform(-0.7854, 0.7854))

   However, I could not determine the range of this random term so that the target object would occasionally not be in the RGB images. How do I ensure that the object exists in the RGB images?

Sorry to ask such a simple question, looking forward to your answer.Thanks!

[themasterlink]

Hey,

1. How to limit the depth values?

data = bproc.renderer.render()
data["depth"] = np.clip(data["depth"], d_min, d_max)

Would this work? You tried to modify the placement of the camera not the actual depth values.

2. How to ensure an object is visible?

There is a check you can use: https://dlr-rm.github.io/BlenderProc/blenderproc.api.camera.html?highlight=visible%20objects#blenderproc.camera.visible_objects

found_objects = bproc.camera.visible_objects(camera_matrix, sqrt_number_of_rays=25)
if your_object in found_objects:
    bproc.camera.add_camera_pose(camera_matrix)

Best, Max

[bjutlss]

Well,thanks for your reply. First,I do not mean that limit the depth of depth map,I want to set max and min for z in cam_t_m2c, The depth of the camera from the target object.

data = bproc.renderer.render()
data["depth"] = np.clip(data["depth"], d_min, d_max)

The above method does not seem to solve my problem. Second, If I want the target object to appear randomly in rgb images, not fixed in the center, is adding a random item the right choice? I just need to add a visual object detection,as following:

        if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.3}, bop_bvh_tree):
            found_objs = bproc.camera.visible_objects(cam2world_matrix, sqrt_number_of_rays=25)
            if sampled_target_bop_objs[0] in found_objs:
                bproc.camera.add_camera_pose(cam2world_matrix, frame=cam_poses)
                cam_poses += 1

It seems to work correctly, just as I thought it would.

[themasterlink]

Hey,

if you want to check that the distance to the object is a certain range, you can use the perform_obstacle_in_view_check:

if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.3, "max": 1.0}, bop_bvh_tree):
    pass

What you posted seems otherwise correct.

Best, Max

[bjutlss]

I have modified the code,like this:

if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.1, "max": 0.6}, bop_bvh_tree):
   found_objs = bproc.camera.visible_objects(cam2world_matrix, sqrt_number_of_rays=25)
       if sampled_target_bop_objs[0] in found_objs:
        bproc.camera.add_camera_pose(cam2world_matrix, frame=cam_poses)
         cam_poses += 1

Trying to limit the range from 0.1m to 0.6m,but in scene_gt.json,the cam_t_m2c still beyond that,as following:

  "75": [{"cam_R_m2c": [0.4270216226577759, 0.8699132800102234, -0.24678681790828705, 0.82950758934021, -0.4855020046234131, -0.2760539650917053, -0.35995832085609436, -0.08683043718338013, -0.9289196133613586], "cam_t_m2c": [174.9066619873047, -141.49192810058594, **1039.97412109375**], "obj_id": 3}],
  "76": [{"cam_R_m2c": [0.7742821574211121, 0.6244383454322815, -0.10278072208166122, 0.5012342929840088, -0.7042668461799622, -0.5027646422386169, -0.38633060455322266, 0.3377644717693329, -0.8582911491394043], "cam_t_m2c": [-186.02288818359375, 43.59092712402344, 422.397705078125], "obj_id": 3}],
  "77": [{"cam_R_m2c": [-0.16214637458324432, 0.8454673886299133, 0.5088164210319519, 0.7354549765586853, 0.44732701778411865, -0.5089251399040222, -0.6578869223594666, 0.29169124364852905, -0.6943356394767761], "cam_t_m2c": [5.1683478355407715, -144.47117614746094, **889.2922973632812**], "obj_id": 3}],
  "78": [{"cam_R_m2c": [0.12128978967666626, 0.9924784302711487, 0.0166047103703022, 0.9873616099357605, -0.11891189962625504, -0.10477331280708313, -0.10201073437929153, 0.02910279855132103, -0.9943578243255615], "cam_t_m2c": [-349.8229675292969, -196.2069549560547, **672.109130859375**], "obj_id": 3}],
  "79": [{"cam_R_m2c": [0.15399110317230225, 0.9807368516921997, 0.120174340903759, 0.4269343912601471, 0.043641597032547, -0.9032288193702698, -0.8910744190216064, 0.19039571285247803, -0.41198983788490295], "cam_t_m2c": [-5.837288856506348, 133.43775939941406, 469.5951232910156], "obj_id": 3}],

PS:My target object is a screw, the models_info as following: "3": {"diameter": 44.262993672518796, "min_x": 0.0, "min_y": -3.5, "min_z": -3.5, "size_x": 44.0, "size_y": 7.0, "size_z": 7.0} }

[themasterlink]

Hey,

sorry why should the distance in the camera matrix matter?

Maybe you should explain why you need to limit this value?

Best, Max

[bjutlss]

Hey, I render these images to train the robot arm with the eye-on-hand RGBD camera. In this case, very long distances seem useless because the eye-on-hand camera cannot reach that height, so I expect to limit the range of objects to the maximum depth of the camera.I expect to render some images within the reach of my robotic workspace.

Best, Max

[themasterlink]

Hey,

So are you trying to limit the distance between the object and the camera or between the camera and the world origin?

Best, Max

[bjutlss]

Hey, I am trying to limit the distance between the object and the camera.

[themasterlink]

Hey,

I have modified the code,like this:

if bproc.camera.perform_obstacle_in_view_check(cam2world_matrix, {"min": 0.1, "max": 0.6}, bop_bvh_tree):
   found_objs = bproc.camera.visible_objects(cam2world_matrix, sqrt_number_of_rays=25)
       if sampled_target_bop_objs[0] in found_objs:
        bproc.camera.add_camera_pose(cam2world_matrix, frame=cam_poses)
         cam_poses += 1

Trying to limit the range from 0.1m to 0.6m,but in scene_gt.json,the cam_t_m2c still beyond that,as following:

  "75": [{"cam_R_m2c": [0.4270216226577759, 0.8699132800102234, -0.24678681790828705, 0.82950758934021, -0.4855020046234131, -0.2760539650917053, -0.35995832085609436, -0.08683043718338013, -0.9289196133613586], "cam_t_m2c": [174.9066619873047, -141.49192810058594, **1039.97412109375**], "obj_id": 3}],
  "76": [{"cam_R_m2c": [0.7742821574211121, 0.6244383454322815, -0.10278072208166122, 0.5012342929840088, -0.7042668461799622, -0.5027646422386169, -0.38633060455322266, 0.3377644717693329, -0.8582911491394043], "cam_t_m2c": [-186.02288818359375, 43.59092712402344, 422.397705078125], "obj_id": 3}],
  "77": [{"cam_R_m2c": [-0.16214637458324432, 0.8454673886299133, 0.5088164210319519, 0.7354549765586853, 0.44732701778411865, -0.5089251399040222, -0.6578869223594666, 0.29169124364852905, -0.6943356394767761], "cam_t_m2c": [5.1683478355407715, -144.47117614746094, **889.2922973632812**], "obj_id": 3}],
  "78": [{"cam_R_m2c": [0.12128978967666626, 0.9924784302711487, 0.0166047103703022, 0.9873616099357605, -0.11891189962625504, -0.10477331280708313, -0.10201073437929153, 0.02910279855132103, -0.9943578243255615], "cam_t_m2c": [-349.8229675292969, -196.2069549560547, **672.109130859375**], "obj_id": 3}],
  "79": [{"cam_R_m2c": [0.15399110317230225, 0.9807368516921997, 0.120174340903759, 0.4269343912601471, 0.043641597032547, -0.9032288193702698, -0.8910744190216064, 0.19039571285247803, -0.41198983788490295], "cam_t_m2c": [-5.837288856506348, 133.43775939941406, 469.5951232910156], "obj_id": 3}],

So, why do you check the camera position in world coordinates? The distance between the object and the camera ranges from 0.1 to 0.6. That has nothing to do with the global position of the camera in world coordinates.

Best, Max