This project implements a simulated grasp-and-lift process in V-REP using the Barrett Hand, with an interface through a python remote API. The goal of this project is to collect information on where to place individual contacts of a gripper on an object. The emphasis on constraining the grasp to specific contacts is to promote future work in learning fine manipulation skills.
This is an extension of a previous project, which largely simplifies the collection process, and introduces domain randomization into the image collection scheme.
Sample data can be viewed Here, which makes use of the first 100 Procedurally Generated Random Objects from Google. For each object, all sub-collision objects were merged into a single entity, rescaled to fit the Barrett Hand, and saved in the meshes/ folder. 10 Grasps were attempted for each object, and images were saved in the .PNG format.
Install the trimesh library:
pip install trimesh==2.29.10
Add the remote API interfaces by copying the following files to lib/:
Open _scenes/graspscene.ttt in V-REP. You should see a table, a few walls, and a Barrett Hand.
Start the scene by clicking the play (triangle) button at the top of the screen. Run the main collection script in src/collect_grasp.py:
cd src
python collect_grasps.py
This will look in the folder data/meshes, and run the first mesh it finds. You should see a mesh being imported into the scene, falling onto the table, then after a short delay a gripper should begin to attempt grasping it. When the gripper closes, it will check whether all the fingertips of the Barrett Hand are in contact with the object - if so, it will attempt to lift the object to a position above the table. Successful grasps (where the object remains in the grippers palm) are recorded and saved in an hdf5 dataset for that specific object in the output/collected folder.
For both pre- and post-grasp, the following information is recorded:
Simulation Property | What's Recorded |
---|---|
Reference frames | palm, world, workspace, object |
Object properties | mass, center of mass, inertia |
Joint angles | All joints of Barrett Hand |
Contacts | positions, forces, and normals |
Images | RGB, depth, and a binary object mask |
Once data collection has finished, we can supplement the dataset with images by indexing the .hdf5 files in the output/collected/ directory:
cd src
python collect_images.py
We first remove potential duplicates grasps from the dataset (based on wrist pose, then contact positions & normals), and remove any extreme outliers. Feel free to modify this file to suit your needs. Next, the simulation will open up or connect to a running V-REP scene, and begin collecting images using data from output/grasping.hdf5. This script uses the state of the simulator at the time of the grasp (i.e. the object pose, gripper pose, angles, etc ...) and restores those parameters before taking an image.
Image randomization is done according to the following properties:
Render Property | What's Affected |
---|---|
Lighting | Number of lights, relative position |
Colour | Random RGB for object and workspace |
Colour Components | Ambient diffuse, specular, emission, auxiliary |
Texture Mapping | Plane, sphere, cylinder, cube |
Texture Pose | Position, Orientation |
Camera | Pose (Resolution, Field of View, near/far planes also supported) |
Data from this phase is stored in the output/processed/ folder, where the *.hdf5 file stores the grasp metadata, and images are stored in the corresponding directory. RGB images of the object, object + gripper are stored in the .jpg format, while the depth image is stored as a 16-bit floating point numpy array.
Sample images of the pre-grasp:
This project is still under construction, so things may still be changing. Additionally, a few points to consider: