Detect and grasp real objects

[CthLeo]

I appreciate the excellent work you guys are doing. With the Dope you have developed, I have been able to detect cans using a custom dataset.

I only used NDDS to generate DR data, which is well detected in the simulation environment, but the detection of real cans is not accurate! The model I am using is very similar to the real cans. I would like to know how to create synthetic data that would allow more accurate detection of real cans?

Also, I can get information about the object pose from rostopic. I understand from your paper that this information is under the camera coordinate system. There is a question of how I can use this information to implement real object grasping, and what is necessary to do away from real grasping Work?

Looking forward to your responses.

[TontonTremblay]

Can you share with me some images of your training data and from where you want to deploy your model?

I think explaining how to move the robot to a specific pose is above the context of this github repo. You should look into move it for example.

[CthLeo]

I made a mistake that I used bowls rather than cans for training. Some examples of my training data are as follows.

I want to detect the bowls placed on the table.

I know that the camera needs to be calibrated to let the robot know the coordinates of the detected object. How to determine the appropriate grasping point based on the detected pose coordinates? I would very appreciate if you have some suggestions.

[TontonTremblay]

You might want to add more variation to your training data, more random background, and more random lights. The problem with the bowl is that it is a symmetrical object. I still have to come up with a clever way to represent these rotations for DOPE.

For grasping just have a fixed point with respect to the object origin and go to that location to close the gripper.

[CthLeo]

Thank you very much for your suggestions. I have created new datasets that add some random lights and random background. The datasets are being trained.

Moreover, I found a problem when testing the network with the testing data generated by NDDS. Through the /dope/rgb_points node published by ros, I can see the accurate bounding box as follows:

However, the "location" predicted by the network is very different from its ground truth, and the debug output is as follows:

It is noted that in the training data and testing data, I used the same camera of NDDS, and the picture resolution was also the same.

I want to know what causes the bounding box to be accurate but the predicted position is very inaccurate. Looking forward to your reply.