search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
352 stars 77 forks source link

Data generation for new tasks described by PDDL #102

Closed rutadesai closed 2 years ago

rutadesai commented 2 years ago

Hi there,

I am interested in using the code in gen folder (PDDL-based expert trajectory generation) for different tasks (e.g., make pizza + cookies) than the ones defined in the alfred paper. I can define these tasks using PDDL as well. Would this be possible? Would you be able to provide a brief description of your gen code including an entry point for the same?

Thanks very much for your help!

MohitShridhar commented 2 years ago

@rutadesai, the simulator (THOR) used by ALFRED has a limited number of objects and interactions. I don't think pizzas and cookies are part of the environment, afaik. But if you are interested in using PDDL to solve general planning problem, then yes, PDDL is applicable. PDDL-based planning typically requires two things: (1) a symbolic description of states (2) a symbolic world model of how you can act in the environment. With this you can use something like FF to plan a sequence of actions to achieve a goal. Checkout this example: https://github.com/guillaume-chevalier/AI-Planning-Solver-Shakeys-World-PDDL

rutadesai commented 2 years ago

Thanks, @MohitShridhar! Sorry I wasn't clear. I am willing to add the necessary objects and actions in AI2Thor for my usecase. And I actually have the pddl domain file and solver working in python with a toy example. I am interested in integrating it in AI2Thor to get corresponding video data of these tasks. So, I was hoping to take advantage of your code for the same. I am wondering if this sounds doable with your gen code and if so what would be some entry points in the code for 1) adding new objects and actions correspond to a pddl domain file 2) changing the pddl solver (I need to use OPTIC solver instead of FF), and 3) way to visualize a single trajectory once I have done #1 and #2.

Thanks very much again! Ruta

MohitShridhar commented 2 years ago

Ah I see, thanks for clarifying. So assuming you have added your objects to THOR 2.1.0 (this commit) and recompiled the binaries with Unity, you can adapt the PDDL domain and problem descriptions to your tasks:

  1. Modify the domain to reflect new actions here.
  2. Add a new list of goals to the library.
  3. Use this function to convert THOR metadata into PDDL problem files.
  4. Then use generate_trajectories.py script to generate your new tasks. This should automatically show you THOR visualizations of agent executions.

Hope this helps!

rutadesai commented 2 years ago

thanks very much, @MohitShridhar!! This is exactly what I was looking for! :)