allenai / ai2thor

An open-source platform for Visual AI.
http://ai2thor.allenai.org
Apache License 2.0
1.09k stars 210 forks source link

Failure to PutObject(s) Lettuce and Bread onto Fridge - no valid positions found #1210

Closed FibonacciDude closed 2 months ago

FibonacciDude commented 2 months ago

Hello,

I'm currently running a 2-agent iThor simulation in FloorPlan3. Initially I have each agent pick up Lettuce and Bread respectively, navigate to the fridge, open the door, and place the objects in the fridge. All the steps are successful except the placing of the objects into the fridge.

The agent is facing the open fridge, and the fridge is interactable. Yet, when any agent tries to place their object w/ "PutObject", the environment says "No valid positions to place object found." Weirdly, this does not happen when putting the exact same objects in the Fridge for FloorPlan1,2.

Screen Shot 2024-04-25 at 1 57 10 PM

When I let ForceAction=True, the action is successful but the objects disappear entirely.

Could the size of the objects be a potential issue? How should we go about resolving this. Many thanks!

winthos commented 2 months ago

Hello! The issue here is there is a built in visibility reliant check related to this action. If you do not set ForceAction=True then the object may fail to find a position if the agent is not able to see the object after the placement. This is why when you set ForceAction=True the object is successfully placed but it disappears from sight, the object is being placed in the free space further back in the fridge.

There are two ways to get around this. You can either position the agent such that it has a better sight of where it is trying to put something, so lowering the camera angle by crouching might be needed. You can also just validate an object is successfully placed via the action metadata directly, even though you won't have direct visual confirmation from the returned visual frames.

Additionally, you may be seeing different behavior across different fridges and scenes because the layouts of the scenes and the size, height, and internal dimensions (free space) of fridges are all very different from each other especially across scenes.

FibonacciDude commented 2 months ago

Yep, I did forceAction=True and the object was indeed just above the agent's eyeline. Thanks!