create new synthetic dataset

[frk1993]

Hi,

I want to create a new synthetic dataset for a setup at university with intel real sense camera. But when I try to run the generate_mask_dataset.py I get many errors. What are the prerequirements to run this file? Can I create a new dataset? I have read, that I need object meshes? Can you provide me your objects meshes? Or How can I get this object meshes?

Best Regards

[frk1993]

Hi, I could find the object meshes and can run the generate_mask_dataset.py. But I am still facing following error : ValueError: No backends available for convex decomposition!

Best regards

[mjd3]

This is actually an error coming from trimesh. To export meshes to URDF, you need to install vhacd, which can be done using the following script: https://github.com/mikedh/trimesh/blob/30a423b884903905aba82408255f02dec0b33175/docker/builds/vhacd.bash. Let me know if you get it working after that!

[frk1993]

Hi, Thank you for the answer. The warnings above are not so important? I installed the vhacd, but now I get following errors: CalledProcessError and qhull input error (see my terminal output). I hope you can help me further.

Traceback (most recent call last): File "/anaconda3/lib/python3.6/site-packages/trimesh/exchange/urdf.py", line 51, in export_urdf convex_pieces = convex_decomposition(mesh, kwargs) File "/anaconda3/lib/python3.6/site-packages/trimesh/decomposition.py", line 18, in convex_decomposition result = _engines[engine](mesh, kwargs) File "/anaconda3/lib/python3.6/site-packages/trimesh/decomposition.py", line 24, in decomposition_automatic result = interfaces.vhacd.convex_decomposition(mesh, **kwargs) File "/anaconda3/lib/python3.6/site-packages/trimesh/interfaces/vhacd.py", line 42, in convex_decomposition result = vhacd.run(_vhacd_executable + argstring) File "/anaconda3/lib/python3.6/site-packages/trimesh/interfaces/generic.py", line 70, in run stderr=subprocess.STDOUT) File "/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/local/bin/testVHACD', '--input', '/var/folders/72/55nkj3991zd1w5m1zngmx2v80000gn/T/tmp3db7dnqw.obj', '--output', '/var/folders/72/55nkj3991zd1w5m1zngmx2v80000gn/T/tmpsaglxt8q.obj', '--log', '/var/folders/72/55nkj3991zd1w5m1zngmx2v80000gn/T/tmp__fw3aue']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "generate_mask_dataset.py", line 279, in generate_segmask_dataset env.reset() File "/Users/frktna/Desktop/Masterarbeit/SDmaskRCNN/sd-maskrcnn/sd_maskrcnn/envs/bin_heap_env.py", line 165, in reset self._reset_state_space() File "/Users/frktna/Desktop/Masterarbeit/SDmaskRCNN/sd-maskrcnn/sd_maskrcnn/envs/bin_heap_env.py", line 85, in _reset_state_space state = self._state_space.sample() File "/Users/frktna/Desktop/Masterarbeit/SDmaskRCNN/sd-maskrcnn/sd_maskrcnn/envs/state_spaces.py", line 427, in sample heap_state = self.heap.sample() File "/Users/frktna/Desktop/Masterarbeit/SDmaskRCNN/sd-maskrcnn/sd_maskrcnn/envs/state_spaces.py", line 215, in sample self._physics_engine.add(workspace_obj, static=True) File "/Users/frktna/Desktop/Masterarbeit/SDmaskRCNN/sd-maskrcnn/sd_maskrcnn/envs/physics_engine.py", line 89, in add trimesh.exchange.export.export_urdf(geometry, urdf_dir) File "/anaconda3/lib/python3.6/site-packages/trimesh/exchange/urdf.py", line 57, in export_urdf convex_pieces = [mesh.convex_hull] File "/anaconda3/lib/python3.6/site-packages/trimesh/caching.py", line 99, in get_cached value = function(*args, **kwargs) File "/anaconda3/lib/python3.6/site-packages/trimesh/base.py", line 1956, in convex_hull hull = convex.convex_hull(self) File "/anaconda3/lib/python3.6/site-packages/trimesh/convex.py", line 53, in convex_hull qhull_options=qhull_options) File "qhull.pyx", line 2359, in scipy.spatial.qhull.ConvexHull.init File "qhull.pyx", line 354, in scipy.spatial.qhull._Qhull.init scipy.spatial.qhull.QhullError: QH7023 qhull warning: unknown 'Q' qhull option n, rest ignored QH6022 qhull input error: 2'th dimension's new bounds [-0.5, 0.5] too wide for existing bounds [ 0, 0]

While executing: | qhull i QbB Pp Qt QJn Options selected for Qhull 2015.2.r 2016/01/18: run-id 570369392 incidence QbBound-unit-box 0.5 Pprecision-ignore Qtriangulate

Best Regards

[mjd3]

I'm guessing that the path you're using to call VHACD is not the same as the one you've installed it at or there is a permissions error somewhere along the way. Can you check to see if the vhacd path is correct? Can you run vhacd without calling it through trimesh? If the error persists but you can call vhacd from the command line, I'd suggest opening a trimesh issue (https://github.com/mikedh/trimesh), as the maintainer of that repo probably will have a better idea than I do of what to try next!

[frk1993]

Thanks for the reply. I opened an issue in trimmest repository and they say that a plane do not have a convex decomposition, and therefore I get this error. But I am just using your code and this works fine for you ? https://github.com/mikedh/trimesh/issues/409

Best regards

[mjd3]

Ah yeah I had not realized you were trying to export the plane. It needs a custom URDF because as mikedh mentioned, there is no convex decomposition for it. I have updated the data directory with a URDF for the plane (see #16). You can move it into your URDF cache directory (specified in generate_mask_dataset.yaml) and it should resolve this problem.

[frk1993]

Thanks. Now I could solve the problem but I get only full grey images as depth_image. I am executing the code on google collaboratory. Could this be a problem? Here the image

And here the output of my execution:

Best Regards

[mjd3]

Can you pull the latest from master and try setting debug to 1 in generate_mask_dataset.yaml? With debug set to 1, you should see a window that pops up and shows the objects falling one by one in slow motion. The depth image that is then generated should show the same scene. Let me know if you are able to see that!

[frk1993]

I will try it now but Is this possible that I get this process simulated when I am using google colabratory?

And another question. I get following warning at the beginning of the execution. Can this cause the problem?

[mjd3]

Ah yeah the Colab environment does not have an active display manager, so you will need to use EGL or OSMesa for offscreen rendering. There are instructions for how to do that here: https://pyrender.readthedocs.io/en/latest/examples/offscreen.html. In addition, there are Colab notebooks in the main readme here: https://github.com/mmatl/pyrender.

I am not concerned by the warnings you've attached; these come from autolab_core and aren't related to the modules we are using.

[frk1993]

Exactly, I am using the OSMesa approach to run your code. As you asked me, I executed your newest code with Debug=1 and I get following Error message. And the code hangs after second Warning. I waited 10 minutes but nothing happens.

[mjd3]

You won't be able to run in debug mode on Colab since it pops a pyrender Viewer instance; sorry I missed that you had said that you were developing on Colab earlier. I will try running on Colab and get back to you as soon as I can!

[frk1993]

Thanks a lot for you help. I am waiting for your message. Here are the most important lines from my colab notebook. By the way I am just using 5 meshes which I downloaded from dexNet homepage http://berkeleyautomation.github.io/dex-net/. But I think the number of meshes should not be a problem?

Once again, thank you very much.

[mjd3]

From what I can tell, this may be an OSMesa issue :(. I tried your setup on Google CoLab and on my local machine and ran into the same issue (blank depth image) on both. I will continue looking into the reason behind this.

The good news is that I was able to get everything running with EGL in a Google CoLab notebook here: https://colab.research.google.com/drive/1iafphvk6oRT_RF0_fD6XwbHw8tpHqZeu. I've also added a script that allows you to download a bunch of meshes from the YCB dataset to get started with the dataset generation much easier. If there is no need for OSMesa in particular, then I'd suggest EGL for now! The fact that the dataset generation works with two of the three pyrender backends but not OSMesa makes me think this is an issue somewhere in pyrender or one of its dependencies. I'll poke around some more there.

[frk1993]

Oh thank you very much. Now its also working for me, the depth images are looking good :) But the semantic mask images are totally black for me. The semantic mask image is created by using the modal mask images? For creating a setup for the intel realsense camera I should only modify the configuration files ? Can you please add a description for these parameters from states.yaml (workspace, center...)?

For object meshes, do I have access to your dataset? I picked some meshes from the dexnet homepage http://berkeleyautomation.github.io/dex-net/

Best Regards

[mjd3]

The semantic masks may not be completely black; the pixel values are very small for the objects (since the background is labeled as 0, the first mask as 1, second mask as 2, etc.). Try visualizing in the same way as I did in the notebook and let me know if you can see them that way!

I have added comments for each parameter in states.yaml. If the main change is to the camera, I'd recommend changing the parameters listed under "camera" to match your resolution, camera intrinsics, distance from the scene, etc.

As for the object mesh dataset, we are working on cleaning it up and should be able to release it at some point! Until then, I'd recommend either sticking with the YCB meshes or creating your own dataset.

[frk1993]

Thanks a lot for your help. I have two further questions, maybe you can help me.

1. I want to use another bin for dataset generation, which I designed with SketchUp tool. But it does not work. What do I have to do, to use my own bin?

2. When benchmarking, you are removing bin pixels to enhance the segmentation results. But this is not possible in real world scenario. Can you suggest me something to enhance the results?

Best regards

[mjd3]

@frk1993, for 1) you should be able to use your own bin by uploading the relevant mesh and pose files to sd-maskrcnn/sd_maskrcnn/data/bin/ (or some other known path) and replacing the lines in the config file here https://github.com/BerkeleyAutomation/sd-maskrcnn/blob/e30438a3ab74efa57e5de652359d3a070934d7e0/cfg/partials/states.yaml#L14 with the names of your files.

For 2), we actually do remove bin pixels in the real images as well! The way we do this right now is by filtering our real depth image either by subtracting a depth image of the empty box or by filtering the point cloud via a rectangular box to get a binary mask that contains all of the non-bin objects in the image. For an example of what these look like, check out the segmasks_filled directory in the wisdom-real dataset. We do remove the masks that are predicted on the bin, since we have knowledge of where it is in the scene and thus it is a simple post-processing step to remove these masks.

Let me know if you have any more questions and thanks for all of your input!

[Sean-Hastings]

Hi, I am running into the same issue as frk1993 ran into earlier. My full terminal output is below, the important part I believe is the SIGABORT from VHACD when called by trimesh. I have a recent (a few days) clone of master, and the urdf cache has a plane folder, but nothing within that folder which may be the issue. Running with debug=1 opens the simulation visualization window but no objects drop in it. Is there some way other than or on top of the included download script to get the plane urdf? alternatively, might this be a different issue?

Thanks!

~/sd-maskrcnn$ python tools/generate_mask_dataset.py datasets/generated
WARNING:root:Failed to import ros dependencies in rigid_transforms.py
WARNING:root:autolab_core not installed as catkin package, RigidTransform ros methods will be unavailable
WARNING:root:Unable to import pylibfreenect2. Python-only Kinect driver may not work properly.
WARNING:root:Failed to import ROS in Kinect2_sensor.py. Kinect will not be able to be used in bridged mode
WARNING:root:Unable to import Primsense sensor modules! Likely due to missing OpenNI2.
WARNING:root:Unable to import pyrealsense2.
WARNING:root:Failed to import ROS in ensenso_sensor.py. ROS functionality not available
WARNING:root:Failed to import ROS in phoxi_sensor.py. PhoXiSensor functionality unavailable.
WARNING:root:Unable to import generic sensor modules!.
WARNING:root:Unable to import weight sensor modules!
pybullet build time: Jun  7 2019 11:24:13
root       INFO     Root logger now logging to datasets/generated/dataset_generation.log
tools/generate_segmask_dataset.py INFO     Creating env took 0.006 sec
tools/generate_segmask_dataset.py INFO     State: 000000
PybulletPhysicsEngine WARNING  Failed to create dir /home/seanhastings/sd-maskrcnn/datasets/objects/urdf/cache/plane. The object may have been created simultaneously by another process
PybulletPhysicsEngine INFO     Exporting URDF for object plane
trimesh    ERROR    problem with convex decomposition, using hull
Traceback (most recent call last):
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/exchange/urdf.py", line 51, in export_urdf
    convex_pieces = convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/decomposition.py", line 21, in convex_decomposition
    return interfaces.vhacd.convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/vhacd.py", line 42, in convex_decomposition
    result = vhacd.run(_vhacd_executable + argstring)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/generic.py", line 70, in run
    stderr=subprocess.STDOUT)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/testVHACD', '--input', '/tmp/tmp6zvhurio.obj', '--output', '/tmp/tmprhqw61e5.obj', '--log', '/tmp/tmpbvbmi23z']' died with <Signals.SIGABRT: 6>.
tools/generate_segmask_dataset.py WARNING  Heap failed!
tools/generate_segmask_dataset.py WARNING  QH7023 qhull warning: unknown 'Q' qhull option n, rest ignored
QH6022 qhull input error: 2'th dimension's new bounds [-0.5, 0.5] too wide for
existing bounds [ 0,  0]

While executing:  | qhull i QbB Qt Pp QJn
Options selected for Qhull 2015.2.r 2016/01/18:
  run-id 1058835860  incidence  QbBound-unit-box 0.5  Qtriangulate
  Pprecision-ignore

Traceback (most recent call last):
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/exchange/urdf.py", line 51, in export_urdf
    convex_pieces = convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/decomposition.py", line 21, in convex_decomposition
    return interfaces.vhacd.convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/vhacd.py", line 42, in convex_decomposition
    result = vhacd.run(_vhacd_executable + argstring)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/generic.py", line 70, in run
    stderr=subprocess.STDOUT)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/testVHACD', '--input', '/tmp/tmp6zvhurio.obj', '--output', '/tmp/tmprhqw61e5.obj', '--log', '/tmp/tmpbvbmi23z']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/generate_mask_dataset.py", line 278, in generate_segmask_dataset
    env.reset()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/bin_heap_env.py", line 158, in reset
    self._reset_state_space()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/bin_heap_env.py", line 84, in _reset_state_space
    state = self._state_space.sample()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/state_spaces.py", line 428, in sample
    heap_state = self.heap.sample()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/state_spaces.py", line 213, in sample
    self._physics_engine.add(workspace_obj, static=True)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/physics_engine.py", line 86, in add
    trimesh.exchange.export.export_urdf(geometry, urdf_dir)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/exchange/urdf.py", line 57, in export_urdf
    convex_pieces = [mesh.convex_hull]
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/caching.py", line 99, in get_cached
    value = function(*args, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/base.py", line 1995, in convex_hull
    hull = convex.convex_hull(self)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/convex.py", line 53, in convex_hull
    qhull_options=qhull_options)
  File "qhull.pyx", line 2359, in scipy.spatial.qhull.ConvexHull.__init__
  File "qhull.pyx", line 354, in scipy.spatial.qhull._Qhull.__init__
scipy.spatial.qhull.QhullError: QH7023 qhull warning: unknown 'Q' qhull option n, rest ignored
QH6022 qhull input error: 2'th dimension's new bounds [-0.5, 0.5] too wide for
existing bounds [ 0,  0]

While executing:  | qhull i QbB Qt Pp QJn
Options selected for Qhull 2015.2.r 2016/01/18:
  run-id 1058835860  incidence  QbBound-unit-box 0.5  Qtriangulate
  Pprecision-ignore

tools/generate_segmask_dataset.py WARNING  None
Traceback (most recent call last):
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/exchange/urdf.py", line 51, in export_urdf
    convex_pieces = convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/decomposition.py", line 21, in convex_decomposition
    return interfaces.vhacd.convex_decomposition(mesh, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/vhacd.py", line 42, in convex_decomposition
    result = vhacd.run(_vhacd_executable + argstring)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/interfaces/generic.py", line 70, in run
    stderr=subprocess.STDOUT)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/testVHACD', '--input', '/tmp/tmp6zvhurio.obj', '--output', '/tmp/tmprhqw61e5.obj', '--log', '/tmp/tmpbvbmi23z']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "tools/generate_mask_dataset.py", line 472, in <module>
    save_tensors=save_tensors, warm_start=warm_start)
  File "tools/generate_mask_dataset.py", line 278, in generate_segmask_dataset
    env.reset()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/bin_heap_env.py", line 158, in reset
    self._reset_state_space()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/bin_heap_env.py", line 84, in _reset_state_space
    state = self._state_space.sample()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/state_spaces.py", line 428, in sample
    heap_state = self.heap.sample()
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/state_spaces.py", line 213, in sample
    self._physics_engine.add(workspace_obj, static=True)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/sd_maskrcnn/envs/physics_engine.py", line 86, in add
    trimesh.exchange.export.export_urdf(geometry, urdf_dir)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/exchange/urdf.py", line 57, in export_urdf
    convex_pieces = [mesh.convex_hull]
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/caching.py", line 99, in get_cached
    value = function(*args, **kwargs)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/base.py", line 1995, in convex_hull
    hull = convex.convex_hull(self)
  File "/home/seanhastings/miniconda3/envs/sd_maskrcnn_fix/lib/python3.7/site-packages/trimesh/convex.py", line 53, in convex_hull
    qhull_options=qhull_options)
  File "qhull.pyx", line 2359, in scipy.spatial.qhull.ConvexHull.__init__
  File "qhull.pyx", line 354, in scipy.spatial.qhull._Qhull.__init__
scipy.spatial.qhull.QhullError: QH7023 qhull warning: unknown 'Q' qhull option n, rest ignored
QH6022 qhull input error: 2'th dimension's new bounds [-0.5, 0.5] too wide for
existing bounds [ 0,  0]

While executing:  | qhull i QbB Qt Pp QJn
Options selected for Qhull 2015.2.r 2016/01/18:
  run-id 1058835860  incidence  QbBound-unit-box 0.5  Qtriangulate
  Pprecision-ignore

[Sean-Hastings]

Followup: I copied the files from the commit you linked and manually placed them in the plane folder and it works great now.

Thank you for this tool!

[mjd3]

Yeah unfortunately I think that's the best solution for the plane URDF for now. Happy to reconsider if you think there's an easier way!

[mjd3]

Closing this issue, as the main issue reported here is actually due to a pyrender/OSMesa issue.

[Viky397]

Hi, I could find the object meshes and can run the generate_mask_dataset.py. But I am still facing following error : ValueError: No backends available for convex decomposition!

Best regards

Hello, where did you find these object meshes? I'm trying to do a similar thing. Thank you.