search

uhh-lt / path2vec

Learning to represent shortest paths and other graph-based measures of node similarities with graph embeddings
Apache License 2.0
33 stars 14 forks source link

How to get the right embeddings for ambiguous words? #28

Closed mbarbouch closed 4 years ago

mbarbouch commented 4 years ago

I want to combine the embeddings of all found word synsets to represent sentence embeddings.

Consider a sentence containing words with multiple meanings. For example, "the customer has opened a bank account". The word "bank" has multiple synsets in WordNet. So how can I get the right one given the context of the sentence?

My guess was to use sentence_wsd() in wsd\graph_wsd_test_v2.py. I adjusted the code a little bit to support any sentence (without having the senseeval annotations), but the result of selected synsets was not really satisfying.

akutuzov commented 4 years ago

Yes, this is the correct script.
But WSD is a difficult task :) As you can see in the paper, one can expect F1 of about 0.5...0.55 for the SensEval and SemEval test sets, using a path2vec model trained on shortest path similarities. Do you observe a significantly lower performance?

mbarbouch commented 4 years ago

Yes, this is the correct script. But WSD is a difficult task :) As you can see in the paper, one can expect F1 of about 0.5...0.55 for the SensEval and SemEval test sets, using a path2vec model trained on shortest path similarities. Do you observe a significantly lower performance?

Thank you. Yes I could reproduce F1-scores (at least close to the values reported).

So what I did next to applying the function to random sentences is to check if the same synsets are selected for the example from the paper shown in Fig. 4, i.e. "More often than not, ringers think of the church as something stuck on the bottom of the belfry".

For this I used the pre-trained embedding models provided on the home page of this repo. I got the following selected nodes:

JCN-S = ['0 more.r.01', '1 frequently.r.01', '2 ringer.n.04', '4 church.n.02', '6 buttocks.n.01', '7 belfry.n.02'] LCH = ['0 more.r.01', '1 frequently.r.01', '2 ringer.n.04', '4 church.n.02', '6 bottom.n.04', '7 campanile.n.01'] ShP = ['0 more.r.01', '1 frequently.r.01', '2 ringer.n.03', '4 church.n.02', '6 bed.n.03', '7 belfry.n.02'] WuP = ['0 more.r.01', '1 frequently.r.01', '2 ringer.n.04', '4 church.n.04', '6 buttocks.n.01', '7 campanile.n.01']

In all four cases we see some differences with selected nodes in Figure 4 (e.g. ShP => '2 ringer.n.03' vs. '2 ringer.n.02'). Since the models are trained on the older version of WordNet, we would expect to get the same selection, right? Or am I missing a point here?

WSD is indeed a difficult task. This is not the focus of my work, however, but I have to handle it somehow :). Your work seems promising, since path2vec has shown better performance in comparison to other popular models like node2vec and Deepwalk.

akutuzov commented 4 years ago

Figure 4 is only an example, given for illustrative purposes (to explain how our WSD approach works). You should not expect that the results on actual models will always be the same.

mbarbouch commented 4 years ago

Figure 4 is only an example, given for illustrative purposes (to explain how our WSD approach works). You should not expect that the results on actual models will always be the same.

Alright. I'll work with it then, with an F1 of ~.53 in mind.

Any suggestions how to improve on this?

I found the following paper "Language Modelling Makes Sense: Propagating Representations through WordNet for Full-CoverageWord Sense Disambiguation" claiming to be state-of-the-art. However, it's not clear yet how to use the method for selecting right synsets. I need to read it more in detail...

akutuzov commented 4 years ago

There exists extensive WSD literature, with lots of stuff coming in recently with contextualized language models. I am sure you will find lots of hints in the ACL Anthology.