search

askforalfred / alfred

ALFRED - A Benchmark for Interpreting Grounded Instructions for Everyday Tasks
MIT License
352 stars 77 forks source link

Inquiry Regarding the Leaderboard #133

Closed SeungyounShin closed 5 months ago

SeungyounShin commented 1 year ago

I am curious to know whether the test server is capable of recognizing attributes such as color, heated, cooled, etc. or relations such as in front of, by, behind, etc.

for unseen tasks like "Place {} on the silver-colored cabinet" or "Place it in front of the coffeemaker"~~ 'Place heated tomato on the table' Is recognized?

thomason-jesse commented 1 year ago

See the paper for full details, but the upshot is "no" for most of these properties. Whether an object has been heated or cooled is checked; those properties are required for several ALFRED tasks. However, spatial information is limited to in/on relationships (e.g., egg in bowl; bowl on count). Colors and relative orientation relations (e.g., "front") are not checked against.