qcr / benchbot

BenchBot is a tool for seamlessly testing & evaluating semantic scene understanding tools in both realistic 3D simulation & on real robots
BSD 3-Clause "New" or "Revised" License
110 stars 12 forks source link

Ground Truth objects not visible in some scenes in passive mode #69

Closed CJuette closed 2 years ago

CJuette commented 2 years ago

Hi again,

playing around with the sequences in passive mode, I noticed that certain ground truth objects are not even visible to the robot in the observations that are provided to the user. Here are examples from miniroom 1 and 2, where I just combined all RGBD point clouds and am showing the ground truth annotations:

miniroom 1 Here the book is just barely visible, the table not at all.

The table is not visible at all.

miniroom 2

I am now wondering - is this intentional? And are these invisible objects taken into account for the evaluation of the passive part of the challenge? The command was simply ./bin/benchbot_run -f --robot carter_omni --env miniroom:1 --task semantic_slam:passive:ground_truth

btalb commented 2 years ago

Thanks for spotting this @CJuette.

We've increased the FOV by decreasing the focal length from 24mm to 11mm. The sensor images should now roughly match those provided before we migrated to Omniverse. Here's what the new starting view for miniroom:1 should look like:

increased_fov

You can get the latest robot definitions by running a benchbot_install.

It is probably worth re-running any challenge submissions you may have already done, as the scores will probably be better.

Let us know if you have any further issues.