How to import URDF robot model and manipulate it?

b-yogesh commented 3 years ago

Hello,

So, I am trying to create a synthetic dataset for Robot pose estimation, and I have a URDF file. But blender does not support importing of URDF files. A plugin called Phobos (https://github.com/dfki-ric/phobos) gives blender the ability to do that, but it can only do it for Blender version 2.79, which I think BlenderProc does not support. If it is then imported, how do I control the robot using BlenderProc (entity manipulator)?

How can I approach this problem and what could be the possible solution?

cornerfarmer commented 3 years ago

Hey @b-yogesh,

I think there are two ways to get BlenderProc to work with Phobos:

There seems to be ongoing effort to make Phobos work with blender 2.9 (https://github.com/dfki-ric/phobos/issues/194#issuecomment-705082430). So maybe you can already make use of their current development and use it together with blender proc
You try to get the current BlenderProc version to work with version 2.79. However, there were quite some API changes when blender 2.8 has been introduced, so you probably need to make small adjustments on a number of places.

For controlling your robot I would advice you to write your own module (its basically just a python class). In this way its easier to implement more advanced behaviour. Also, it seems to me that the loaded robots are controlled via bones, whose manipulation is not yet supported by the entity manipulator.

b-yogesh commented 3 years ago

Hey @cornerfarmer ,

Thank you very much for your prompt reply. I was trying your first suggestion yesterday (Phobos work with blender 2.9). So, I was able to link Phobos + Blender 2.90 + BlenderProc with a few changes in the script of Phobos. I am able to load the URDF file and manipulate using Rotate function of Blender GUI. It all seemed fine until I ran the basic example of BlenderProc.

This is the image from the Blender RenderImage: blenderrender

And this is the one that BlenderProc generates:

Based on ablation study of the URDF model, when I removed the links of the robot, it gave me the exact image as the BlenderProc. This is the URDF robot structure:

Check out these logs:

Here, it does not load any Links into BlenderProc which causes all the meshes to collapse to the origin, which I suspect is the issue. Can you please help me out here as how do I make this work?

cornerfarmer commented 3 years ago

Hey @b-yogesh,

thanks for the detailed description.

The log you added only contains the rgb renderer, I am not sure if links are explicitly listed here. However, you are probably right, that something in the loading of your robot goes wrong. Can you please explain how you load your URDF model using BlenderProc. Did you write your own module? Which functions from the Phobos extension do you call?

b-yogesh commented 3 years ago

Hello,

Phobos has a built-in functionality to import URDF models into it. I used that.

Have I made any mistake? Because, while importing from phobos, the links are imported and I am able to manipulate them using rotation. But, the links do not appear in the BlenderProc output 😞

Here are the complete logs:

_Blender 2.90.0 (hash 0330d1af29c0 built 2020-08-31 10:00:13) Read prefs: C:\Users\Yogesh Kumar\AppData\Roaming\Blender Foundation\Blender\2.90\config\userpref.blend found bundled python: J:\SaarlandUniversity\MasterThesis\BlenderProc\blender-2.90.0-windows64\2.90\python Importing phobos IMPORT: phobos.defs Parsing definitions from: C:\Users\Yogesh Kumar\AppData\Roaming\phobos/definitions defaultControllers.yml C:\Users\Yogesh Kumar\AppData\Roaming\Blender Foundation\Blender\2.90\scripts\addons\phobos\defs.py:201: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. tmpyaml = yaml.load(evaluateString(tmpstring)) defaultJoints.yml defaultMaterials.yml defaultModel.yml defaultMotors.yml defaultSensors.yml defaultSubmechanisms.yml defaultSubmodels.yml Creating new definition type: joints IMPORT: phobos.display IMPORT: phobos.io Registered mesh plugin: ['obj', 'stl', 'dae'] ERROR in scenes/init: "mars.py" has no valid scene plugin interface. Registered scene plugin: ['sdf world'] Registered scene plugin: ['smurfs'] Registered entity plugin: ['generic'] ERROR in entities/init: "heightmap.py" has no valid entity plugin interface. Registered entity plugin: ['light'] Registered entity plugin: ['primitive'] Registered entity plugin: ['sdf'] Registered entity plugin: ['smurf'] Registered entity plugin: ['srdf'] Registered entity plugin: ['submechanisms'] Registered entity plugin: ['thumbnails'] Registered entity plugin: ['urdf'] Registered entity plugin: ['yaml'] Registered mesh plugin: ['obj', 'stl', 'dae'] ERROR in scenes/init__: "mars.py" has no valid scene plugin interface. Registered scene plugin: ['sdf world'] Registered scene plugin: ['smurfs'] RELOAD: phobos.model IMPORT: phobos.operators IMPORT: phobos.phobosgui RELOAD: phobos.phoboslog RELOAD: phobos.phobossystem RELOAD: phobos.utils Registering operators.selection... Registering operators.io... Registering operators.editing... Registering operators.naming... Registering operators.misc...

Registering phobosgui... ... successful. Info: Deleted 3 object(s) Info: Deleted 3 object(s)

Start - Initializing module main.Initializer

Finished - Initializing module main.Initializer (took 0.014 seconds)

Start - Initializing module loader.BlendLoader

Finished - Initializing module loader.BlendLoader (took 0.004 seconds)

Start - Initializing module lighting.LightLoader

Finished - Initializing module lighting.LightLoader (took 0.011 seconds)

Start - Initializing module camera.CameraLoader

Finished - Initializing module camera.CameraLoader (took 0.003 seconds)

Start - Initializing module renderer.RgbRenderer

Finished - Initializing module renderer.RgbRenderer (took 0.006 seconds)

Start - Initializing module writer.Hdf5Writer

Start - Initializing module postprocessing.TrimRedundantChannels

Finished - Initializing module postprocessing.TrimRedundantChannels (took 0.002 seconds)

Finished - Initializing module writer.Hdf5Writer (took 0.249 seconds)

Start - Running blender pipeline

Start - Running module Initializer

Device GeForce 940MX (Display) of type OPTIX found and used.

Finished - Running module Initializer (took 0.093 seconds)

Start - Running module BlendLoader

Finished - Running module BlendLoader (took 0.591 seconds)

Start - Running module LightLoader

Finished - Running module LightLoader (took 0.000 seconds)

Start - Running module CameraLoader

Finished - Running module CameraLoader (took 0.003 seconds)

Start - Running module RgbRenderer

Finished - Running module RgbRenderer (took 26.805 seconds)

Start - Running module Hdf5Writer

Merging data for frame 0 into J:\SaarlandUniversity\MasterThesis\BlenderProc\BlenderProc\examples/basic/output\0.hdf5 Key: colors - shape: (720, 1280, 3) - dtype: uint8 - path: C:\Users\YOGESH~1\AppData\Local\Temp\blender_proc_81324f8d050e499b8d041559319a78b4\rgb_0000.png Key: distance - shape: (720, 1280) - dtype: float64 - path: C:\Users\YOGESH~1\AppData\Local\Temp\blender_proc_81324f8d050e499b8d041559319a78b4\distance_0000.exr Key: normals - shape: (720, 1280, 3) - dtype: float64 - path: C:\Users\YOGESH~1\AppData\Local\Temp\blender_proc_81324f8d050e499b8d041559319a78b4\normals_0000.exr

Finished - Running module Hdf5Writer (took 3.201 seconds)

Finished - Running blender pipeline (took 30.705 seconds)

Blender quit_

MartinSmeyer commented 3 years ago

So as BlenderProc currently does not know how to apply these link poses, one way would be to write a custom Manipulator that loads the link pose information using

https://github.com/dfki-ric/phobos/blob/cd5357cf3d7fe5bab50ac9b4e4c53fdcc06d7dab/phobos/model/models.py#L160

and then sets these as poses of the visual/inertial meshes.

You could orient yourself at the Entity Manipulator: https://github.com/DLR-RM/BlenderProc/blob/master/src/manipulators/EntityManipulator.py https://github.com/DLR-RM/BlenderProc/tree/master/examples/entity_manipulation

cornerfarmer commented 3 years ago

Hey @b-yogesh,

regarding your loading problem: thanks for the explanation, I missed the info that you save your loaded robot to a .blend file and then import it via the BlendLoader. At the moment we are appending the blend file item-by-item which seems to destroy connections between them and which in the end leads to the results you got. As a temporary workaround you could replace the run function in the BlendLoader (src/loader/BlendLoader.py) with the following code:

    def run(self):
        # get a path to a .blend file
        path = Utility.resolve_path(self.config.get_string("path"))

        with bpy.data.libraries.load(path) as (data_from, data_to):
            data_to.collections = [name for name in data_from.collections]

        for collection in data_to.collections:
            bpy.context.scene.collection.children.link(collection)

This should now load the whole content of your blend file without destroying any links between them. If you then want to further manipulate specific links after loading, you can take a look at what Martin wrote above.

b-yogesh commented 3 years ago

Thank you very much. It worked like a charm! 😄

I just have a few more follow-up questions and would appreciate if you give me some advise.

1) How do I rotate the arms using python at random different angles for each joint (I tried extracting bpy.data.objects[obj_id] and then using the rotate one joint of the arm by some degrees in some axis, but this does not respect the constraints added via URDF. Although, while using the Rotate tool, the constraints are respected). Am I missing something here (i was told that entity manipulator cannot work with robots yet)?

2) To generate a dataset, do I need to write a loop for run.py ?

3) The output of BlenderProc is a HDF file, can I change the script so that it saves only the images (rgb, seg, depth)?

Thank you

themasterlink commented 3 years ago

Hey,

these are great questions, but to make the issues better searchable, please do open new issues for your questions 2 and 3.

Best, Max

b-yogesh commented 3 years ago

Sure, will do!

cornerfarmer commented 3 years ago

@b-yogesh Regarding your first question:

I think you need to change the rotation of the respective bone to make use of its constraints. Changing the rotation should look like bpy.data.objects['Back_x_Link'].pose.bones["Bone"].rotation_euler[1] = 0.1.

With Back_x_Link being the name of your armature object and Bone being the name of the bone you want to manipulate. The rotation you set should now be clipped to the configured limit interval.

shreeramsigdel77 commented 3 years ago

You can simply generate a animation in blender (frame by frame) and I hope you can achieve the random effect. Or you can simply use object pose and manipulate the arms of a robot. My recommendation, creating a animation inside blender.

b-yogesh commented 3 years ago

Thank you for your prompt replies and help. I got it working with this bpy.data.objects['Back_x_Link'].pose.bones["Bone"].rotation_euler[1] = 0.1.

hetolin commented 3 years ago

@b-yogesh Hi, I am trying to import the URDF file with Phobos. But I cannot import the urdf file since it throw some errors when I used the Phobos add-on. Known from your reply that you had a few changes in the script of Phobos. Would you please share more details about the revised part of the Phobos to be compatibility with Blender 2.90?

b-yogesh commented 3 years ago

@b-yogesh Hi, I am trying to import the URDF file with Phobos. But I cannot import the urdf file since it throw some errors when I used the Phobos add-on. Known from your reply that you had a few changes in the script of Phobos. Would you please share more details about the revised part of the Phobos to be compatibility with Blender 2.90?

Hi @hetolin, sorry for the late reply. Yes, so I had used this code from the Phobos older releases for defLayers

def defLayers(layerlist):
    """Returns a list of 20 elements encoding the visible layers according to layerlist
    Args:
      layerlist: 
    Returns:
    """
    if not isinstance(layerlist, list):
        layerlist = [layerlist]
    layers = 20 * [False]
    for layer in layerlist:
        layers[layer] = True
    return layers

Paste this code in the following directory phobos/utils/blender.py. Remove the installation.conf and run the phobos setup again.

Hope this helps

hetolin commented 3 years ago

@b-yogesh hi, thank you for your detailed reply. But after following the instructions as you provided. I still met a problem. The blender throws the error: `Python: Traceback (most recent call last): File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute models.buildModelFromDictionary(model) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary model['links'][lnk]['object'] = linkmodel.createLink(link) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 125, in createLink geometrymodel.createGeometry(vis, 'visual', newlink) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 150, in createGeometry bpy.context.scene.layers = bUtils.defLayers(defs.layerTypes[geomsrc]) AttributeError: 'Scene' object has no attribute 'layers'

location: :-1`

Is the bpy.context.scene.layers no longer used in Blender 2.9?

b-yogesh commented 3 years ago

Hi @hetolin, can you try commenting this line bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) line (150) in /home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py

hetolin commented 3 years ago

@b-yogesh I tried your instruction, and it also throws the error: (I am using Blender 2.91, Maybe I need to change to Blender2.90? Have you meet such a problem when you are using phobos in Blender?)

Python: Traceback (most recent call last): File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute models.buildModelFromDictionary(model) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary model['links'][lnk]['object'] = linkmodel.createLink(link) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 125, in createLink geometrymodel.createGeometry(vis, 'visual', newlink) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 151, in createGeometry bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) TypeError: bpy_struct: item.attr = val: Window.view_layer expected a ViewLayer type, not list location: :-1

b-yogesh commented 3 years ago

Hello @hetolin, even I am using blender 2.91, and it works just fine. Although, I would like to ask, which branch of phobos have you installed? You should install this branch https://github.com/dfki-ric/phobos/tree/blender2.9 and not the master branch

Also, I did not see this line of code bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) Line 151 in the geometries.py. I think you might have added it by yourself and forgot to remove it. Try removing that, and restart blender. Let me know how that goes

hetolin commented 3 years ago

Hello, @b-yogesh I have installed the blender2.9 branch of phobos. And I comment bpy.context.window.view_layer = bUtils.defLayers(defs.layerTypes[geomsrc]) Line 151. It throws new errors:

Python: Traceback (most recent call last): File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/operators/io.py", line 283, in execute models.buildModelFromDictionary(model) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/models.py", line 1208, in buildModelFromDictionary model['links'][lnk]['object'] = linkmodel.createLink(link) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/links.py", line 127, in createLink geometrymodel.createGeometry(col, 'collision', newlink) File "/home/hetolin/.config/blender/2.91/scripts/addons/phobos/model/geometries.py", line 204, in createGeometry newgeom.select = True AttributeError: 'Object' object has no attribute 'select'

location: :-1

and I comment newgeom.select = True(line 204) it seems works now.

b-yogesh commented 3 years ago

Good to hear that it works now :). So, in blender, in the scene collection, you have one collection called 'Collision' or 'Collision joints'. I would recommend disabling that (click on the eye symbol). For coloring, you can add material to each joint. Let me know how it goes.

hetolin commented 3 years ago

Thank you for your guidance. After disabling the 'Collision' collection. It works now. thank you for your detailed instructions

b-yogesh commented 3 years ago

Glad that it worked out. :)

hetolin commented 3 years ago

@b-yogesh hello, I have successfully imported the urdf file in the blender. But the 'visual' model shown in the blender(top row picture)is not visually consistent with that shown in the Rviz(bottom row picture). Noted that I used the same urdf file. Have you ever had such a problem?

themasterlink commented 2 years ago

Hey,

we now have URDF support in BlenderProc ;)

Best, Max

DLR-RM / BlenderProc