when using my own data and test it with the demo code, no result or pointcloud shows

[madinwei]

Hello, Again THANK YOU for this AWSOME repository.

I tried to use my own data from my VZense camera. which can produce depth and IR images. after I look through the code I understand that I need to change my Camera's intrinsic matrix according to mine so I change the fx,fy, cx, and cy values in the intrinsic.npy.

However, the result has 0 predictions and there is no pointcloud in the mayavi window.

I hope anyone can guide me or give any suggestions on it, or if I have missed something.

[yiyeChen]

@madinwei

Thanks for your interest! I am out of the lab this summer so cannot reply on time. Sorry for the inconvenience.

If there is 0 predictions, I suggest reducing the threshold and/or looking at the 2D keypoint plot to check the detected keypoint. In the test script this is done via these lines. I can add the visualization feature to the demo script too if you needed, but might need some extra time as I am out this summer. If no reasonable keypoint group shows up even if you have reduced the threshold, then perhaps the domain gap between your data and ours is too large - what kind of data are you testing on?

It is abnormal to have no point cloud showing up in the mayavi window. Even if there is no detections, the scene point cloud should appear. Can you double check the image data you input, especially the depth map? I have not used your camera before, but if the D435 camera is not well-calibrated, there might be many zero value pixels in the depth map, which are filtered out in my visualization function.

[madinwei]

@yiyeChen Am very grateful for your reply and help, and I apologize for disturbing you during your summer vacation.

While I was trying to solve my problem, I tried to normalize my point cloud, and then the mayavi window showed my pointcloud with the prediction. as in these screenshots: when normalized : without normalizing:

However, with the normalized pointcloud the prediction grasp will have different x,y, and z values. and their location will not represent the real world. do you have any suggestions in this matter? how to get the value to represent the real world like the non-normalize pointcloud.

If there is 0 predictions, I suggest reducing the threshold and/or looking at the 2D keypoint plot to check the detected keypoint. In the test script this is done via these lines. I can add the visualization feature to the demo script too if you needed, but might need some extra time as I am out this summer. If no reasonable keypoint group shows up even if you have reduced the threshold, then perhaps the domain gap between your data and ours is too large - what kind of data are you testing on?

do you mean with the domain gap the range of the values of the pointcloud? is this why my normalized point cloud worked, since the range of the values is smaller?

It is abnormal to have no point cloud showing up in the mayavi window. Even if there is no detections, the scene point cloud should appear. Can you double check the image data you input, especially the depth map? I have not used your camera before, but if the D435 camera is not well-calibrated, there might be many zero value pixels in the depth map, which are filtered out in my visualization function.

I think it appears but since the pointcloud values range is big, each point is far from its neighbor. this is my understanding, I am still a novice in many things and learning.

again I am very grateful for your reply and help, I am Looking forward to your response.

[yiyeChen]

@madinwei

While I was trying to solve my problem, I tried to normalize my point cloud, and then the mayavi window showed my pointcloud with the prediction. as in these screenshots:

I am not certain I follow. Since our method processes the RGB-D image rather than the point clouds, normalizing point clouds or not shouldn't have any impact on the grasp prediction. In fact they are only used for visualization to confirm the predictions. So I am not sure why you only have predictions after the normalization - previously by the result has 0 predictions, do you simply mean cannot see any predictions in the mayavi window?

I think it appears but since the pointcloud values range is big, each point is far from its neighbor.

It is possible! In this case you can simply use the scrollwheel to zoom in or zoom out.

[madinwei]

@yiyeChen . I appreciate your reply and help in your vacation time.

I am not certain I follow. Since our method processes the RGB-D image rather than the point clouds, normalizing point clouds or not shouldn't have any impact on the grasp prediction. In fact they are only used for visualization to confirm the predictions. So I am not sure why you only have predictions after the normalization - previously by the result has 0 predictions, do you simply mean cannot see any predictions in the mayavi window?

I did the normalizing step to the depth image, which will be fed to the KGN, as in this screenshot: If I do not use any normalization, I will not get any predictions result or can't see my data in the mayavi window.

It is possible! In this case you can simply use the scrollwheel to zoom in or zoom out.

I understand, but because the point is far from each other I can't see anything even if I zoom out, but I think if the point size in mayavi window is big I might see the pointcloud.

[yiyeChen]

Okay I see. It is possible that your depth value range is way beyond our training distribution, hence our network cannot predict any result? As shown here, our camera is at most 1 meter away from the table center in our training, hence the depth range is also around this value. But according to your screen shot, your depth could be 6000 (mm)? That is why normalization is critical to get results.

You can consider (a) convert your depth to meter unit. (b) if you still cannot get any result, just normalize the depth, and then normalize the grasp pose back to your scale. To do that, you can simply multiply the predicted translation and open width with your normalization scale.

[madinwei]

Discussing with you is very fruitful, and I appreciate it.

OH, mine is in cm, It slipped my mind, I will convert the values to m and also try to normalize the predicted grasp pose to my scale.

many thanks to you.