I have been struggling for a while now to get the GraspNerf Model to output a valid TSDF Volume for my real world Images.
This is the network output of both the Quality volume and the TSDF Volume which has mostly missing values.
Another Image of just the TSDF Volume with mostly missing values:
I also get the error: !! too low ratio tensor([0.0000, 0.0000, 0.0811, 0.3172, 0.0142, 0.0000], device='cuda:0') from the sample_volume function inside "renderer.py".
This is the Pointcloud I get when I plot the depth images from the Realsense camera along with their corresponding extrinsincs and intrinsics. It looks fine and the bottle object I have placed inside the Task Frame is in the right position.
These are my extrinsics from the camera positions placed on a hemisphere with radius 0.5m and polar angle pi/6 around the middle of the workspace. Also plotted are the extrinsics from the simulated images which rotate around a fixed point 0.5m above the origin.
Here I don't understand why the z-axis of these extrinsics does not point to the middle of the workspace but instead in a seemingly random direction?
This is the TSDF Volume from the model fed with simulated images from the run_simgrasp demo. It looks fine and is centered in the workspace even though their extrinsics don't point this way.
I have formatted my real images, extrinsics and intrinsics in exactely the same format as the simulated ones and also tried cropping and resizing to shape (288, 512) but this does not improve my results, they look almost identical.
I would be really grateful if somebody could help me out here and explain to me where I am wrong in my thinking. I am happy to supply any additional information if required.
I have been struggling for a while now to get the GraspNerf Model to output a valid TSDF Volume for my real world Images.
This is the network output of both the Quality volume and the TSDF Volume which has mostly missing values.
Another Image of just the TSDF Volume with mostly missing values:
I also get the error:
!! too low ratio tensor([0.0000, 0.0000, 0.0811, 0.3172, 0.0142, 0.0000], device='cuda:0')from the sample_volume function inside "renderer.py".This is the Pointcloud I get when I plot the depth images from the Realsense camera along with their corresponding extrinsincs and intrinsics. It looks fine and the bottle object I have placed inside the Task Frame is in the right position.
These are my extrinsics from the camera positions placed on a hemisphere with radius 0.5m and polar angle pi/6 around the middle of the workspace. Also plotted are the extrinsics from the simulated images which rotate around a fixed point 0.5m above the origin. Here I don't understand why the z-axis of these extrinsics does not point to the middle of the workspace but instead in a seemingly random direction?
This is the TSDF Volume from the model fed with simulated images from the run_simgrasp demo. It looks fine and is centered in the workspace even though their extrinsics don't point this way.
I have formatted my real images, extrinsics and intrinsics in exactely the same format as the simulated ones and also tried cropping and resizing to shape (288, 512) but this does not improve my results, they look almost identical.
I would be really grateful if somebody could help me out here and explain to me where I am wrong in my thinking. I am happy to supply any additional information if required.