PDDL Trajectory Generation in ProcTHOR

pushkalkatara commented 1 year ago

Hi, Thanks for the amazing dataset.

I wanted to generate PDDL-based expert demonstrations in ProcTHOR dataset similar to ALFRED-gen dataset.

I modified the Layout generation script to support the ProcTHOR-10k dataset.

However, I think the AI2THOR nightly build which supports ProcTHOR does not support receptacles. While applying the PutObject action in layout generation, I get this error:

  File "/home/anaconda3/envs/procthor/lib/python3.7/site-packages/ai2thor/controller.py", line 983, in step
    raise ValueError(self.last_event.metadata["errorMessage"])
ValueError: 
        Action: "PutObject" called with invalid argument: 'receptacleObjectId'
        Expected arguments: String objectId, Boolean forceAction = False, Boolean placeStationary = True, Int32 randomSeed = 0
        Your arguments: 'objectId', 'receptacleObjectId', 'forceAction', 'placeStationary'
        Valid ways to call "PutObject" action:
                Void PutObject(String objectId, Boolean forceAction = False, Boolean placeStationary = True, Int32 randomSeed = 0)
                Void PutObject(Single x, Single y, Boolean forceAction = False, Boolean placeStationary = True, Int32 randomSeed = 0, Boolean putNearXY = False)

Is there a way I can fix this issue for PDDL-based data generation in ProcTHOR?

MohitShridhar commented 1 year ago

@pushkalkatara, if ProcTHOR doesn't support receptacles or object interactions, then most of the ALFRED code+dataset might not be useful for you. You need the PDDL planner if you want solve tasks that involve interacting with the environment, but otherwise for navigation, a simple A* planner would be sufficient.

Hope this helps!

pushkalkatara commented 1 year ago

@MohitShridhar Thanks for the prompt response. I think procTHOR does support object interactions, it also has a baseline for rearrangement. Probably the design of how interaction (PutObject) is supported is different in procTHOR (v=ai2thor-nightly) than Alfred (v=ai2thor-2.1). It does have PutOn X,Y coordinate support:

Void PutObject(Single x, Single y, Boolean forceAction = False, Boolean placeStationary = True, Int32 randomSeed = 0, Boolean putNearXY = False)

but doesn't support receptacleObjectId. I've raised an issue on the procTHOR-10k repo as well. If the nightly adds support for receptacles, that would allow port PDDL to procTHOR easily.

MohitShridhar commented 1 year ago

oh cool! yeah, you probably have to followup with procTHOR folks on this. The 2.1.0 version we used for ALFRED is almost 3 years old, so it probably won't gel with the new API.

pushkalkatara commented 1 year ago

Sure, thanks for the suggestions.

pushkalkatara commented 1 year ago

Hi @MohitShridhar,

I made changes with respect to the new ai2thor API supported by procTHOR, the data generation code seems to work now.

I generated a few layouts using the modified script, and trying to generate data with the modified generate_trajectories script.

I am able to generate trajectories for some tasks like:

Type: train Row:  63574 Scene House0 seed 6
problem id 63574
Task: put fork in a bowl and then put them in countertop
Performing reset via thor_env API
Resetting ProcThorEnv
Performing restore via thor_env API
> /home/sirdome/EmbodiedProject/ET/alfred/gen/generate_trajs.py(617)generate()
-> agent.step(action_dict)
(Pdb) action_dict
{'action': 'Plan'}
(Pdb) next
planned /home/ET/data/logs_gen/planner/generated_problems/problem_63574.pddl in 0.14762, plan length 8 solver type 5
planned /home/ET/data/logs_gen/planner/generated_problems/problem_63574.pddl in 0.14800, plan length 8 solver type 4
planned /home/ET/data/logs_gen/planner/generated_problems/problem_63574.pddl in 0.15935, plan length 8 solver type 3
plan
000: GotoLocation loc: loc|17|3|1|30
001: PickupObject Fork|surface|2|17_copy_11     Full: {'objectId': 'Fork|surface|2|17_copy_11', 'receptacleObjectId': 'Fridge|2|1'}
002: GotoLocation loc: loc|17|3|1|15
003: PutObject o: Fork|surface|2|17_copy_11 r: Bowl|surface|2|11
004: PickupObject Bowl|surface|2|11     Full: {'objectId': 'Bowl|surface|2|11', 'receptacleObjectId': 'Fridge|2|1'}
005: GotoLocation loc: loc|4|10|3|60
006: PutObject o: Bowl|surface|2|11 r: CounterTop|2|0
007: Action: End        Full: {'value': 1}

However, there are many failure cases like:

Empty Plans


PDDLGoalType: pick_cool_then_place_in_recep
Saving images to: /home/data/ET_procthor/pick_cool_then_place_in_recep-Cup-None-DiningTable-53/trial_T20221027_062700_730936/raw_images/
Resetting ProcThorEnv
set seed in game_state_base reset 797390
setup random goal ----------------------------------------------
seed None
info ('train', 78078)
--------------------------------------------------------------------

Type: train Row: 68398 Scene House53 seed 838797390 problem id 68398 Task: put a cool cup in diningtable Performing reset via thor_env API Resetting ProcThorEnv Performing restore via thor_env API {'action': 'Plan'} plan 000: Action: End Full: {'value': 0} Traceback (most recent call last): File "generate_trajs.py", line 605, in generate event = agent.step(action_dict) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/deterministic_planner_agent.py", line 18, in step super(DeterministicPlannerAgent, self).step(action) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/semantic_map_planner_agent.py", line 56, in step self.plan_agent.execute_plan() File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/plan_agent.py", line 25, in execute_plan raise ValueError('Empty plan is successful, no work to do') ValueError: Empty plan is successful, no work to do Error: ValueError('Empty plan is successful, no work to do') Invalid Task: skipping... %%%%%%%%%% errors (2): (1.00) (2) Empty plan is successful, no work to do %%%%%%%%%%

2. LookAction failed, Point Teleport Failure (not sure if [this](https://github.com/askforalfred/alfred/issues/98) is related to this issue)

sampled tuple: ('pick_two_obj_and_place', 'SaltShaker', 'None', 'CounterTop', 53) PDDLGoalType: pick_two_obj_and_place Saving images to: /home/sirdome/EmbodiedProject/ET/data/ET_procthor/pick_two_obj_and_place-SaltShaker-None-CounterTop-53/trial_T20221027_063331_964989/raw_images/ Resetting ProcThorEnv set seed in game_state_base reset 887156 setup random goal ---------------------------------------------- seed None info ('train', 84810)

Type: train Row: 60000 Scene House53 seed 274887156 problem id 60000 Task: find two saltshaker and put them in countertop Performing reset via thor_env API Resetting ProcThorEnv Performing restore via thor_env API {'action': 'Plan'} planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_60000.pddl in 0.23544, plan length 10 solver type 5 planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_60000.pddl in 0.23632, plan length 9 solver type 3 planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_60000.pddl in 0.23834, plan length 9 solver type 4 plan 000: GotoLocation loc: loc|13|10|1|30 001: PickupObject SaltShaker|surface|2|1_copy_22 Full: {'objectId': 'SaltShaker|surface|2|1_copy_22'} 002: GotoLocation loc: loc|3|15|0|60 003: PutObject o: SaltShaker|surface|2|1_copy_22 r: CounterTop|2|0 004: GotoLocation loc: loc|13|9|1|30 005: PickupObject SaltShaker|surface|2|1 Full: {'objectId': 'SaltShaker|surface|2|1'} 006: GotoLocation loc: loc|3|15|0|60 007: PutObject o: SaltShaker|surface|2|1 r: CounterTop|2|0 008: Action: End Full: {'value': 1} /home/sirdome/anaconda3/envs/procthor/lib/python3.7/site-packages/ai2thor/server.py:395: UserWarning: The key event.metadata["reachablePositions"] is deprecated and has been remapped to event.metadata["actionReturn"]. 'The key event.metadata["reachablePositions"] is deprecated and has been remapped to event.metadata["actionReturn"].' step action Action: RotateLeft step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} nav done step action PickupObject SaltShaker|surface|2|1_copy_22 Full: {'objectId': 'SaltShaker|surface|2|1_copy_22', 'forceVisible': True} step action Action: RotateLeft step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: RotateLeft step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} step action Action: MoveAhead Full: {'moveMagnitude': 0.25} Point teleport failure step action Action: RotateRight Point teleport failure Traceback (most recent call last): File "generate_trajs.py", line 605, in generate event = agent.step(action_dict) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/deterministic_planner_agent.py", line 18, in step super(DeterministicPlannerAgent, self).step(action) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/semantic_map_planner_agent.py", line 56, in step self.plan_agent.execute_plan() File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/plan_agent.py", line 38, in execute_plan self.controller_agent.step(plan_action, executing_plan=True) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/deterministic_planner_agent.py", line 18, in step super(DeterministicPlannerAgent, self).step(action) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/semantic_map_planner_agent.py", line 53, in step self.game_state.gt_graph.navigate_to_goal(self.game_state, start_pose, end_pose) File "/home/sirdome/EmbodiedProject/ET/alfred/gen/graph/graph_obj.py", line 345, in navigate_to_goal raise Exception('Look action failed %s' % event.metadata['errorMessage']) Exception: Look action failed Error: Exception('Look action failed ') Invalid Task: skipping...



Are these expected to fail? How should I go about debugging these? 
Any suggestions would be really helpful. Thanks!

thomason-jesse commented 1 year ago

We utilized rejection sampling during the generation of ALFRED trajectories. Some scene initializations just aren't compatible with the sampled task (e.g., if you're supposed to put a fork in a cup, but there's already a fork in a cup by random object starting positions). Other times navigation fails, objects are out of reach, etc. Very expected for many generated trajectories to fail before completion. You can twiddle with the PDDL and execution to increase the success rate, but the generated scenes will sometimes just have conditions that can't be overcome (e.g., you need a particular plate, but the plate is on a low shelf where the agent can't see/reach from any angle).

pushkalkatara commented 1 year ago

Thanks for clearing out. How should I go about tweaking PDDL and execution to increase the success rate?

What would be a baseline success rate? I'm basically trying to understand if the failure roots to scene configuration, object placements, simulator-API parameters, or something else.

I tested on a single procthor Kitchen environment. Stats were:

Successful: 88 Failed: 289 Success rate ~ 30% Task Distribution : {'pick_two_obj_and_place': 27, 'pick_and_place_simple': 44, 'pick_and_place_with_movable_recep': 17}.

The trajectories didn't have task distributions related to:

         "pick_clean_then_place_in_recep",
         "pick_heat_then_place_in_recep",
         "pick_cool_then_place_in_recep",
         "look_at_obj_in_light",

Most of these tasks were failing due to Point teleport failure.

one such instance:

Type: train Row:  57048 Scene House53 seed 989664915
problem id 57048
Task: put a cool winebottle in countertop
Performing reset via thor_env API
Resetting ProcThorEnv
Performing restore via thor_env API
{'action': 'Plan'}
planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_57048.pddl in 0.04053, plan length 8 solver type 5
planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_57048.pddl in 0.04147, plan length 7 solver type 3
planned /home/sirdome/EmbodiedProject/ET/data/logs_gen/planner/generated_problems/problem_57048.pddl in 0.04149, plan length 7 solver type 4
plan
000: GotoLocation loc: loc|17|4|0|60
001: PickupObject WineBottle|surface|2|23       Full: {'objectId': 'WineBottle|surface|2|23'}
002: GotoLocation loc: loc|6|10|3|60
003: Action: CoolObject Full: {'receptacleObjectId': 'Fridge|2|1'}
004: GotoLocation loc: loc|3|15|0|60
005: PutObject o: WineBottle|surface|2|23 r: CounterTop|2|0
006: Action: End        Full: {'value': 1}
/home/sirdome/anaconda3/envs/procthor/lib/python3.7/site-packages/ai2thor/server.py:395: UserWarning: The key event.metadata["reachablePositions"] is deprecated and has been remapped to event.metadata["actionReturn"].
  'The key event.metadata["reachablePositions"] is deprecated and has been remapped to event.metadata["actionReturn"].'
step action Action: RotateRight
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
step action Action: MoveAhead   Full: {'moveMagnitude': 0.25}
Point teleport failure
step action Action: RotateRight
Point teleport failure
Traceback (most recent call last):
  File "generate_trajs.py", line 606, in generate
    event = agent.step(action_dict)
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/deterministic_planner_agent.py", line 18, in step
    super(DeterministicPlannerAgent, self).step(action)
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/semantic_map_planner_agent.py", line 56, in step
    self.plan_agent.execute_plan()
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/plan_agent.py", line 38, in execute_plan
    self.controller_agent.step(plan_action, executing_plan=True)
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/deterministic_planner_agent.py", line 18, in step
    super(DeterministicPlannerAgent, self).step(action)
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/agents/semantic_map_planner_agent.py", line 53, in step
    self.game_state.gt_graph.navigate_to_goal(self.game_state, start_pose, end_pose)
  File "/home/sirdome/EmbodiedProject/ET/alfred/gen/graph/graph_obj.py", line 345, in navigate_to_goal
    raise Exception('Look action failed %s' % event.metadata['errorMessage'])
Exception: Look action failed 
Error: Exception('Look action failed ')

thomason-jesse commented 1 year ago

You're integrating a newer version of AI2THOR, so your guess is going to be as good as ours for how to go about fixing bugs as they arise. I can see some deprecation warning in the transcript you provided; that's probably a reasonable place to start. We're not going to have useful info about rejection sampling rates since you're working in a new space with a new version, though. This extension is a cool idea, but it's out of the scope of "maintaining the ALFRED benchmark codebase", which is what we still do here :)

pushkalkatara commented 1 year ago

Yes, I am basically trying to backtrack through scene configurations in procthor scenes which might be a probable reason for pddl plan failure. Probably the failures relate to how objects are arranged in a scene and the agent's visibility. this might give some direction. thanks for the suggestions.

Regarding the success rate, I was referring to PDDL plans success rate in alfred scenes just to get a raw idea of % how many generated plans are executed in the environment.

askforalfred / alfred

PDDL Trajectory Generation in ProcTHOR #125