use Chatgpt for WSD

[johan456789]

From my preliminary experiments, it seems to do quite well on WSD. I think it's a good idea to try using ChatGPT to generate definitions.

Click to see ChatGPT generated definition

 

• [x] a) experiment using dictionary definitions and ChatGPT to select the matching one
  • [x] WordNet senses
  • [x] Merriam Webster
• [ ] b) #25
• [ ] decide whether to use a) or b)

[johan456789]

a) experiment using dictionary definitions and ChatGPT to select the matching one

Using ChatGPT considerably improves the accuracy

For example,

Simple Lesk got this wrong:

ChatGPT got this right:

But there are also words that both Simple Lesk and ChatGPT got wrong:

The correct definition is the second in wordnet:

If we use chatgpt to generate the definition directly, then the definition is correct:

But this method is susceptible to hallucination.

Problems

Wrong stemming from Kindle

One problem caused by wrong stemming from the Kindle. The correct stem here should be 'don'. Wrong definitions got returned and both got this wrong:

The definition field is empty here because wn.synset('globe-trotting') returns nothing. What we need is wn.synset('globe-trot'). Interestingly, simple_lesk returns the correct definition here. So maybe we should just we the lemmatizer in pywsd used by simple_lesk.

Potential solution: use pywsd or nltk's lemmatizer instead of relying on Kindle's.

Incomplete WordNet definitions

Another bigger problem imo is that wordnet definitions are often not good. For example:

"Japan was far from fielding workable guided missiles, but the project gave Morita a glimpse of the future."

None of WordNet's definition match:

Merrium-Webster Learner's does have this:

So does OED:

From this preliminary test, the quality and completeness of the dictionary are in this order: OED >= Merriam-Webster > WordNet. Since Merriam-Webster provides a free API, we should experiment with it first. the completeness of vocabulary OED > WordNet > Merriam-Webster (learner's). Since I think Merriam-Webster's definition is more suitable for language learners, I will use it first and fallback to WordNet if a definition is not found.

[johan456789]

Here's the log when running manually with chatgpt_wsd and merriam-webster dictionary:

2023-07-26-chatgpt_wsd-mw.rtf.zip (WSD accuracy report is wrong in the file #22)

The end result is WSD accuracy is 31 / 41 = 75.6%. This is a pretty good accuracy. A lot of failures are caused by lemmatization. Two possible solutions:

• [x] lemmatize the word using pywsd or other tools before calling m-w API
• [x] change the prompt to chatgpt and ask it to pick the closest definition instead of returning -1