Open lichtefeld opened 4 years ago
A potential solution to this would be to add some kind of attribute for relations not to translate into language, either to Situation
or as a part of the input to the language generator. For the example above, we could do something like:
RuntimeError: Error while generating English for situation {
E salient objects:
E ball
E cookie
E other objects:
E book
E table
E mom[is-speaker[binary,perceivable]]
E learner[is-addressee[binary,perceivable]]
E always relations:
E in-region(ball, Region(cookie,distance=proximal,direction=+FacingAddresseeAxis(_object=cookie)))
E in-region(cookie, Region(ball,distance=proximal,direction=-FacingAddresseeAxis(_object=cookie)))
E relations_to_not_translate = [in-region(ball, Region(cookie,distance=proximal,direction=+FacingAddresseeAxis(_object=cookie)))]
E from template: preposition-training-ball-behind-cookie
E }
so that we get "the cookie is near the ball" rather than trying to translate both. Something similar could work for "over" and "under" where the mirrored relation is added to the relations that shouldn't be translated to language. This is a bit hackish though, and it might be better to fix this in the language generator itself.
To fix this issue in the language generator itself, we could either use some sort of syntax hint or relations_to_not_translate
attribute to directly deal with this, or we could be very conscious of the ordering of the relations and only translate the first one. Since we iterate through the always_relations
to translate them in order, it would make sense to add some kind of condition to catch the fact that a relation between the first and second slots has already been translated and then not try to translate the second one.
This issue already has a PR open for it which begins the work of this change see: https://github.com/isi-vista/adam/pull/457
When we add regional relationships to out scenes we expect to add symmetric relationships to the situation because the perception just copies over the situation relationships remapping the objects. However, when we implement the change which creates the symmetric relationships the English language generator is no longer able to determine how to create a sentence for the scene. For example:
One would be able to assert "a ball behind a cookie" or "a cookie in front of a ball" from this situation. To resolve this issue, we need to give the English language generator (or any language generator) a hint as to what sentence we want to return for the situation.