search

allenai / allenact

An open source framework for research in Embodied-AI from AI2.
https://www.allenact.org
Other
302 stars 48 forks source link

Integration with AllenNLP #261

Open aleSuglia opened 3 years ago

aleSuglia commented 3 years ago

Problem

AllenAct represents a framework for learning to act in 3D environments. Learning complex tasks from scratch, without human supervision is very challenging. Additionally, learning from raw pixels is even harder. Language can represent a relevant medium for improving the sample efficiency of learning to act models. Therefore, I believe that a learning to act framework should be also tightly coupled with a learning to understand language framework.

Desired solution

I would like to see a tight integration between AllenNLP and AllenAct. Particularly, the natural language processing component I believe could be essential for performing specific embodied tasks requiring language understanding.

Do you think this is a reasonable feature to consider? If yes, do you have a plan for it? Happy to contribute to this effort :)

Lucaweihs commented 3 years ago

Hi @aleSuglia,

I think this is a fantastic suggestion. One of the reason we haven't prioritized this yet is that the most popular embodied tasks (e.g. instruction following for navigation) don't seem to require extremely deep language understanding. I see you've done some interesting work with guessing games, is there a task you're interested in where you feel like a tight AllenNLP+AllenAct integration would be very helpful?

aleSuglia commented 3 years ago

Hi @Lucaweihs,

Thanks for your answer. The kind of tasks that I have in mind are those in which agents can interact with environment as well as in natural language with other agents to achieve a given task (e.g. https://arxiv.org/abs/2011.08277). I believe that AllenAct combined with AllenNLP would be the perfect solution to tackle this sort of embodied and communicative tasks. I would be more than happy to start drafting a proposal for this with your help!

aleSuglia commented 3 years ago

Any more thoughts on this?