isi-vista / adam

Abduction to Demonstrate an Articulate Machine
MIT License
10 stars 4 forks source link

Make `far` and `near` implement symmetric relationships for situation generation #456

Open lichtefeld opened 4 years ago

lichtefeld commented 4 years ago

This issue already has a PR open for it which begins the work of this change see: https://github.com/isi-vista/adam/pull/457

When we add regional relationships to out scenes we expect to add symmetric relationships to the situation because the perception just copies over the situation relationships remapping the objects. However, when we implement the change which creates the symmetric relationships the English language generator is no longer able to determine how to create a sentence for the scene. For example:

RuntimeError: Error while generating English for situation {
E               salient objects:
E                   ball
E                   cookie
E               other objects:
E                   book
E                   table
E                   mom[is-speaker[binary,perceivable]]
E                   learner[is-addressee[binary,perceivable]]
E               always relations:
E                   in-region(ball, Region(cookie,distance=proximal,direction=+FacingAddresseeAxis(_object=cookie)))
E                   in-region(cookie, Region(ball,distance=proximal,direction=-FacingAddresseeAxis(_object=cookie)))
E               from template: preposition-training-ball-behind-cookie
E           }

One would be able to assert "a ball behind a cookie" or "a cookie in front of a ball" from this situation. To resolve this issue, we need to give the English language generator (or any language generator) a hint as to what sentence we want to return for the situation.

paynesa commented 4 years ago

A potential solution to this would be to add some kind of attribute for relations not to translate into language, either to Situation or as a part of the input to the language generator. For the example above, we could do something like:

RuntimeError: Error while generating English for situation {
E               salient objects:
E                   ball
E                   cookie
E               other objects:
E                   book
E                   table
E                   mom[is-speaker[binary,perceivable]]
E                   learner[is-addressee[binary,perceivable]]
E               always relations:
E                   in-region(ball, Region(cookie,distance=proximal,direction=+FacingAddresseeAxis(_object=cookie)))
E                   in-region(cookie, Region(ball,distance=proximal,direction=-FacingAddresseeAxis(_object=cookie)))
E               relations_to_not_translate = [in-region(ball, Region(cookie,distance=proximal,direction=+FacingAddresseeAxis(_object=cookie)))]
E               from template: preposition-training-ball-behind-cookie
E           }

so that we get "the cookie is near the ball" rather than trying to translate both. Something similar could work for "over" and "under" where the mirrored relation is added to the relations that shouldn't be translated to language. This is a bit hackish though, and it might be better to fix this in the language generator itself.

To fix this issue in the language generator itself, we could either use some sort of syntax hint or relations_to_not_translate attribute to directly deal with this, or we could be very conscious of the ordering of the relations and only translate the first one. Since we iterate through the always_relations to translate them in order, it would make sense to add some kind of condition to catch the fact that a relation between the first and second slots has already been translated and then not try to translate the second one.