search

YujiaBao / Distributional-Signatures

"Few-shot Text Classification with Distributional Signatures" ICLR 2020
https://arxiv.org/abs/1908.06039
MIT License
254 stars 57 forks source link

data split and experiment result #16

Closed Wangpeiyi9979 closed 4 years ago

Wangpeiyi9979 commented 4 years ago

hi， thanks for your nice work. I have a question for data split, in FewRel, s your partition method the same as [1] partition?

Is the report result of your paper reproduced by yourself? In my own practice, I divide the FewRel dataset according to the method in [1], with ProtoCNN in 5way 1shot, the accuracy rate can reach 76%, I'm really confused. Can you help me, tthx.

[1] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. In Proceedings ofthe 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4803–4809, 2018.

rmwu commented 4 years ago

Hello! Thank you for your question. FewRel was split using a "hard split," in which train classes are distant to val and test to simulate word-level distributional shift. Please see A.4 of our paper (page 15) for more details. In addition, our data splits are provided in the download.

Wangpeiyi9979 commented 4 years ago

Do you mean that you have re-divided the data set and hope that the training set, verification set, and test set will have different word-level distribution? Then have you conducted your experiment on the original data set division? How is the comparison effect with your ProtoCNN

rmwu commented 4 years ago

All results were reported on our splits, including baseline models, which we implemented ourselves. The details behind the exact data splitting algorithms may be found in Appendix A.4.

Wangpeiyi9979 commented 4 years ago

thx