search

asappresearch / structshot

Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning
MIT License
58 stars 18 forks source link

Reproduce performance of structshot #6

Open nguyennth opened 3 years ago

nguyennth commented 3 years ago

Hi,

Thank you for publishing the source code. I have access to OntoNotes and try to reproduce the results in your paper.

In the case of 1-shot learning with NNshot, I could obtain the same average score, i.e., 33.2, although the scores of each dataset are different from yours. However, I'm struggling with Structshot. My scores on CoNLL is much lower than yours, resulting in a lower average score on the three datasets.

In the case of 5-shot learning, I couldn't obtain as high scores as yours both with NNshot and Structshot. In average, Structshot produced worse performance than NNshot.

I wonder if you use a different set of parameters than the one you publish here. I hope you can help me to shed some light here.

Thank you very much for your time!

Best regards, Nhung

yi-asapp commented 3 years ago

Hi Nhung,

Thanks for your interest in our work. For OntoNotes, did you use the splits noted here: https://cemantix.org/conll/2012/data.html ?

The most important parameter for StructShot is tau, which you can tune here: https://github.com/asappresearch/structshot/blob/main/structshot/run_pred.sh#L22

It is a sensitive one, so I would suggest to try values like [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1]

nguyennth commented 3 years ago

Hi Yi,

Thank you very much for your prompt responses.

I use the splits mentioned here: https://cemantix.org/data/ontonotes.html, which I think is probably the same as the splits that you shared. Anyway, I'll double-check them.

Thanks for your suggestions of tau values!