The parsing of sentences to ShapeWorld semantic space

[furkanbiten]

I will just copy and paste what I wrote on previous issue just to keep it as an anchor:

I downloaded the latest commit on analyzer and here are some weird behaviour I am getting, maybe you can help me with those. For example, when I am running analyze = analyzer.Dmrs_analyzer(), analyze([''A pentagon is not smaller than a triangle.'']), I am getting [None].

I will try to find some pattern in the sentences that cause this. Let's see if I can find it.

[furkanbiten]

Alright, I think I got something. If I give to analyzer this caption: 'a yellow shape is a different shape from a pentagon shape'. It return None. However, If I give this caption: 'a yellow shape is a different shape from a pentagon', it parses to Existential object.

More or less similar behaviour is observed if I delete the word "shape" after the color (returning None).

What do you think?

PS: I think I will write some post-processing script to solve this problem, it will be hacky tho, is there a way that is more robust?

[AlexKuhnle]

Unfortunately the analyzer does not more than reverse the generation process, so any variation in the language is likely to not be parsed correctly. It would be possible to extend this grammar-based analysis to cover more of the space of simple "ShapeWorld statements" (ultimately an open-ended process, of course), but that wasn't my focus during the PhD, so I didn't do more. A grammar framework like ERG can be useful, but it's also feasible to do it from scratch for this narrow domain -- it's definitely not just a simple template-matching problem, though, if you want to do it properly. :-)

[AlexKuhnle]

I did a little bit of testing, and you may have found the type of adjective that is not working (for reasons I don't know yet, maybe just a simple fix), where many other sentences seem to work alright, as long as you stick to sentence patterns as they could also have been produced by the generator.

[furkanbiten]

it's definitely not just a simple template-matching problem, though, if you want to do it properly. :-)

To me at this point, it seems like simple template-matching but that is because it is just an idea rather than an actual code existing so I will take your word for it.

so any variation in the language is likely to not be parsed correctly.

Ok, this is very good to know.

A grammar framework like ERG can be useful, but it's also feasible to do it from scratch for this narrow domain.

I will check ERG. What do you mean by from scratch?

[furkanbiten]

I did a little bit of testing, and you may have found the type of adjective that is not working (for reasons I don't know yet, maybe just a simple fix), where many other sentences seem to work alright, as long as you stick to sentence patterns as they could also have been produced by the generator.

Fingers crossed! :D

[AlexKuhnle]

It is a bit like template matching, but given your example sentence involving "not", you probably want it to be compositional (you could but probably quickly decide not to define a template for every sentence with and without not :-) . And once you're starting making it compositional, you need to keep track of some things like the "number" of an object and propagate it accordingly (a square "is" vs two squares "are"). The nice thing about the ERG (the grammar framework behind the ACE/DRMS stuff) is that it takes care of these things and lets you work on an abstracted graph instead. However, it can be a bit annoying to work with the framework, so if you want to extend the "analyzer" capability, I would probably recommend building your own little compositional template-like analysis from scratch. I don't think it's extremely difficult or so, but I'm sure you'll find it's also not quite just a matter of template matching. :-)

[furkanbiten]

I see. Now that you give couple of concrete examples, I see the difficulty in creating template matching. I will be sure to check ERG grammar. If I manage to create something that is compatible with ShapeWorld, I will hit you up if perhaps you would want to integrate as well.

Thank you very much for the direction by the way.

(Most educational github issue :))

[AlexKuhnle]

Sure, I would be happy to add an extension of the analyzer capability. :-)

Also, don't get me wrong, I don't want to discourage building a parser functionality from scratch -- I think it's definitely possible. In fact, unless you know a lot about the (D)MRS framework, it's probably the best way to go about it.

Are you still interested in the sentence example you mentioned above, or is the current analyze functionality anyway too limited for you?

[furkanbiten]

Ok, so let's focus one dataset type, namely relational, then find a way to generalize to all the rest. Given a sentence S, I want to parse it into (Obj_i, Rel_j, Obj_k) where Obj_i is the ith object referred in the sentence, Rel_j is jth relation that is used, and Obj_k is the kth object. As an example, a sentence "A blue shape is below an ellipse" -> ([1,2,3], y-rel, [3,4,5]), where 1,2,3,4,5 is the id of the objects in world. If this can be parsed in this way, imo it has several advantages for evaluation.

So first of all, grammar shouldn't affect the truth value of a sentence, i.e. if the grammar is broken, for example you use "are" instead of "is", that is fine since this will be caught by ACE grammar and you can have a number for this (as you had in the paper, metric for grammar). So the difference between grammar and truthfulness is "completely" seperated (maybe there is some counterexample that I am not seeing).

Second of all, I think this is more in line and easier to use with ShapeWorld, meaning caption.agreement() would easily be used for this type of triplets.

Now, regarding the generalizing to other dataset types, I see it more or less in line with selection type but instead of triplets, i guess pairs need to be used. And for the quantification, I see examples that fit and I see some that doesn't. And to be honest, I didn't give much thought how this could be extended on quantification dataset type, yet.

I hope it was more or less clear.

[AlexKuhnle]

Yes, makes sense, and I agree that it fits well with ShapeWorld and is easier to use. The Caption objects in ShapeWorld are independent of any DMRS/ACE stuff, so you can create the corresponding Caption (e.g. Relation) object from this analysis. That's how I would do it as well. Plus, as you say, you disentangle correctness from grammaticality (which in the paper could be seen as not quite perfect, since correctness required grammaticality, but what's right and not is a bigger linguistic discussion, I'm sure :-).

It will only get a bit more difficult once you get to the various ways of negating, connectives can be interesting, relative clauses may be nontrivial, etc. At that point you'll probably have to move to a more intelligent compositional pattern matching approach, instead of full-sentence template matching. (But it's not that these things are trivially taken care of by the ERG/DMRS framework... it took me a while to get it right.)