search

stepjam / RLBench

A large-scale benchmark and learning environment.
https://sites.google.com/corp/view/rlbench
Other
1.03k stars 218 forks source link

Loading demos at different resolutions causes incorrect data #208

Open beneisner opened 7 months ago

beneisner commented 7 months ago

If you create demos with rendering say, at 256x256 resolution, and then load them using https://github.com/stepjam/RLBench/blob/master/rlbench/utils.py#L41 with CameraConfigs with different sizes (i.e. 128x128),

then all images (including RGB + Depth) are scaled like so: https://github.com/stepjam/RLBench/blob/master/rlbench/utils.py#L198

However, whenever you scale an image, the intrinsics also have to change (for instance, the offset will be totally different). However, the same intrinsics are being used: https://github.com/stepjam/RLBench/blob/master/rlbench/utils.py#L260

This causes the images to look OK, but the point clouds to be totally incorrect.

A fix would transform the intrinsics as well before reconstructing the point cloud.

I don't have time to make a PR for this right now, but wanted to open the issue in case anyone runs into the same behavior. I'm currently getting around it by making sure I always load at the same resolution.