search

isi-vista / adam

Abduction to Demonstrate an Articulate Machine
MIT License
10 stars 4 forks source link

Learning Dynamic Prepositions #769

Closed lichtefeld closed 4 years ago

lichtefeld commented 4 years ago

A selection of our prepositions are truly dynamic meaning that it is difficult to express them outside of the context of an action (i.e. 'toward').

CURRENT LIST TO LEARN:

We will test that our current sub-set verb learner is capable of learning the individual prepositions with the verbs they appear with.

When we have an integrated learner we need to be able to isolate which objects in our scene are the two relevant to the relationship being expressed by the preposition, or the theme. For the sentence "a bird moves a box towards a door" - the learner needs to be able to identify whether the theme is the bird, box or door. While English word order from the non-dynamic prepositions might help us identify that door is a part of the preposition, we might struggle to identify if 'box' or 'bird' should be selected as the theme as both do follow the same SpatialPath of the action.

If the above issue can be resolved then we believe our relation learner should be capable of making a good attempt at the pattern for the preposition independent of the verb.

(For reference see my comments on #748 )

lichtefeld commented 4 years ago

If we try to run our current verb subset learner over the entire dynamic preposition curriculum, knowing that we won't learn the individual relationships but rather a variance of each verb with the attached prepositional information, we are unable to learn at all. Why? In trying to do pattern relaxation for beside we at some point in pattern reduction eliminate a variable_object_slots in the pattern causing slot_3 to no longer exist.

Here's the console log dump from the failure.

[2020-05-18 11:41:00] INFO:root:Observation 15: a dog pushes a box beside a table
[2020-05-18 11:41:00] INFO:root:Object recognizer recognized: [('ground',), ('dog',), ('box',), ('table',), ('car',)]
[2020-05-18 11:41:00] INFO:root:object matching: ms in success: 4831.08033426106, ms in failed: 396.00267447531223
[2020-05-18 11:41:00] INFO:root:Learner observing LanguageAlignedPerception(language=[(a/det, dog/noun, pushes/verb, a/det, box/noun, beside/adp, a/det, table/noun), tree=DependencyTree(_graph=<networkx.classes.digraph.DiGraph object at 0x7f212269af90>, root=pushes/verb, tokens=i{dog/noun, a/det, box/noun, a/det, table/noun, a/det, beside/adp, pushes/verb})], perception_graph=PerceptionGraph(nodes=[gravitational-up[-curved, +directed, +aligned_to_gravity], south-to-north[-curved, +directed, -aligned_to_gravity], west-to-east[-curved, +directed, -aligned_to_gravity], learner, learner-vertical[-curved, +directed, +aligned_to_gravity], learner-left-to-right[-curved, +directed, -aligned_to_gravity], learner-back-to-front[-curved, +directed, -aligned_to_gravity], (Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278917904), (Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278918800), (Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278917776) and 40 more], edges=[(learner, learner-vertical[-curved, +directed, +aligned_to_gravity], primary-axis), (learner, learner-left-to-right[-curved, +directed, -aligned_to_gravity], has-axis), (learner, learner-back-to-front[-curved, +directed, -aligned_to_gravity], has-axis), (learner, (Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278916624), in-region), (learner, (is-addressee[binary,perceivable], learner), has-property), (learner, (is-learner, learner), has-property), (learner, (animate[binary], learner), has-property), (learner, (self-moving[binary], learner), has-property), (learner, (aboutSameSizeAsLearner, learner), has-property), (learner-back-to-front[-curved, +directed, -aligned_to_gravity], learner, facing-axis), ((Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278917904), gravitational-up[-curved, +directed, +aligned_to_gravity], +_GravitationalAxis()), ((Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278917904), MatchedObjectNode(name=('ground',)), reference-object), ((Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278918800), gravitational-up[-curved, +directed, +aligned_to_gravity], +_GravitationalAxis()), ((Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278918800), MatchedObjectNode(name=('ground',)), reference-object), ((Region(the ground,distance=exterior-but-in-contact,direction=+_GravitationalAxis()), 139780278917776), gravitational-up[-curved, +directed, +aligned_to_gravity], +_GravitationalAxis()) and 54 more]), node_to_language_span=i{MatchedObjectNode(name=('dog',)): [0:2), MatchedObjectNode(name=('box',)): [3:5), MatchedObjectNode(name=('table',)): [6:8)}, language_span_to_node=i{[0:2): MatchedObjectNode(name=('dog',)), [3:5): MatchedObjectNode(name=('box',)), [6:8): MatchedObjectNode(name=('table',))}, aligned_nodes=i{MatchedObjectNode(name=('dog',)), MatchedObjectNode(name=('box',)), MatchedObjectNode(name=('table',))})
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsOntologyNodePredicate(property_value=hollow[binary])
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsOntologyNodePredicate(property_value=hollow[binary])]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsOntologyNodePredicate(property_value=self-moving[binary])
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsOntologyNodePredicate(property_value=self-moving[binary])]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsOntologyNodePredicate(property_value=biggerThan)
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsOntologyNodePredicate(property_value=biggerThan)]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsOntologyNodePredicate(property_value=biggerThan)
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsOntologyNodePredicate(property_value=biggerThan)]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsColorNodePredicate(color=#000000)
[2020-05-18 11:41:00] INFO:root:Deleting extra color nodes: [IsColorNodePredicate(color=#000000), IsColorNodePredicate(color=#000000), IsColorNodePredicate(color=#000000)]
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsColorNodePredicate(color=#000000), IsColorNodePredicate(color=#000000), IsColorNodePredicate(color=#000000), IsColorNodePredicate(color=#000000)]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsColorNodePredicate(color=#dbbf21)
[2020-05-18 11:41:00] INFO:root:Deleting extra color nodes: [IsColorNodePredicate(color=#dbbf21)]
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsColorNodePredicate(color=#dbbf21), IsColorNodePredicate(color=#dbbf21)]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is IsColorNodePredicate(color=#0000ff)
[2020-05-18 11:41:00] INFO:root:Deleting extra color nodes: [IsColorNodePredicate(color=#0000ff)]
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [IsColorNodePredicate(color=#0000ff), IsColorNodePredicate(color=#0000ff)]
[2020-05-18 11:41:00] INFO:root:Relaxation: last failed pattern node is MatchedObjectPerceptionPredicate()
[2020-05-18 11:41:00] INFO:root:Nodes to delete directly: [MatchedObjectPerceptionPredicate()]
[2020-05-18 11:41:00] INFO:root:Relaxation: deleted due to disconnection: [IsOntologyNodePredicate(property_value=inanimate[binary]), IsColorNodePredicate(color=#6e5f13), IsOntologyNodePredicate(property_value=biggerThan)]

The second to last line is an invalid removal because removing the MatchedObjectPerceptionPredicate node means we won't be able to realign the English.

gabbard commented 4 years ago

@LichMaster98 : Is this problem still extant or do we expect it to be improved by any of our recent changes?

lichtefeld commented 4 years ago

@gabbard This should be improved by our recent changes. I haven't tried to run the Integrated Relation Learner over any curriculum other than the same curriculum the original learners were run over.

lichtefeld commented 4 years ago

Crashes at observation 140

[2020-07-13 12:45:23] INFO:root:Observation 140: a dog goes to a cookie
[2020-07-13 12:45:23] INFO:root:Object recognizer recognized: [('ground',), ('dog',), ('cookie',), ('table',)]
[2020-07-13 12:45:23] INFO:root:object matching: ms in success: 77603.38945500553, ms in failed: 3089.8627983406186
[2020-07-13 12:45:23] INFO:root:Observation 140: a dog goes to a cookie
Traceback (most recent call last):
  File "/nas/home/jacobl/miniconda3/envs/adam/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/nas/home/jacobl/miniconda3/envs/adam/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/experiment/run_m13.py", line 73, in <module>
    parameters_only_entry_point(main)
  File "/nas/home/jacobl/projects/coref-alternatives-root/repos/vistautils/vistautils/parameters_only_entrypoint.py", line 43, in parameters_only_entry_point
    args=sys.argv[1:],
  File "/nas/home/jacobl/projects/coref-alternatives-root/repos/vistautils/vistautils/parameters_only_entrypoint.py", line 73, in _real_parameters_only_entry_point
    main_method(params)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/experiment/run_m13.py", line 69, in main
    log_experiment_entry_point(experiment_params)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/experiment/log_experiment.py", line 115, in log_experiment_entry_point
    "log_hypothesis_every_n_steps", default=250
  File "/nas/home/jacobl/projects/adam-root/adam/adam/experiment/__init__.py", line 169, in execute_experiment
    LearningExample(perceptual_representation, linguistic_description)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/integrated_learner.py", line 132, in observe
    self.action_learner.learn_from(current_learner_state)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/template_learner.py", line 286, in learn_from
    self._learning_step(preprocessed_input, thing_whose_meaning_to_learn)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/subset.py", line 208, in _learning_step
    for previous_pattern_hypothesis in previous_pattern_hypotheses
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/subset.py", line 209, in <listcomp>
    for hypothesis_from_current_perception in hypotheses_from_current_perception
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/verbs.py", line 265, in _update_hypothesis
    for previous_slot, node1 in previous_pattern_hypothesis.template_variable_to_pattern_node.items()
  File "/nas/home/jacobl/projects/adam-root/adam/adam/learner/perception_graph_template.py", line 148, in intersection
    trim_after_match=trim_after_match,
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/perception_graph.py", line 1001, in intersection
    trim_after_match=trim_after_match,
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/perception_graph.py", line 1284, in relax_pattern_until_it_matches
    initial_partial_match=partial_match,
  File "/nas/home/jacobl/miniconda3/envs/adam/lib/python3.7/site-packages/more_itertools/more.py", line 138, in first
    return next(iter(iterable))
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/perception_graph.py", line 1393, in _internal_matches
    initial_partial_match=merged_initial_partial_match,
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/_matcher.py", line 552, in subgraph_isomorphisms_iter
    self.initialize(initial_partial_match=initial_partial_match)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/_matcher.py", line 204, in initialize
    self._jump_to_partial_match(initial_partial_match)
  File "/nas/home/jacobl/projects/adam-root/adam/adam/perception/_matcher.py", line 266, in _jump_to_partial_match
    f"Requested to begin matching from an alignment which aligns "
RuntimeError: Requested to begin matching from an alignment which aligns semantically infeasible nodes: ObjectSemanticNodePerceptionPredicate() to ObjectSemanticNodePerceptionPredicate()

This error looks like it's coming from the slot alignment choice for jumping to a match.

lichtefeld commented 4 years ago

After fixing the curriculum we can learn these just fine. m13-verbs-with-dynamic-prepositions.zip