search

hadifar / stc_clustering

66 stars 31 forks source link

Unable to reproduce results even after doing the pre-training step for each dataset. #8

Open rajat-tech-002 opened 3 years ago

rajat-tech-002 commented 3 years ago

image I used the same code and did pre-training for each dataset properly. Also, ran the code for 5 steps and took mean and std as mentioned in the paper. Unable to reproduce results for Search Snippets and Biomedical Dataset.

zkharryhhhh commented 3 years ago

image I used the same code and did pre-training for each dataset properly. Also, ran the code for 5 steps and took mean and std as mentioned in the paper. Unable to reproduce results for Search Snippets and Biomedical Dataset.

Hello, thanks for your sharing. I have the same problem , the results I reproduces are similar to yours. For Search Snippets, acc is 69.71, nmi is 54.23. For Biomedical, acc is 35.87, nmi is 30.17. I do experiments many times, the results for these two datasets have a slight change ,which are all much worse than the results in paper. Can you sovle the problem now?maybe the settings of hyperparameter? Can you give some advice?

hadifar commented 3 years ago

Thanks for your interest in our paper. As you already might notice, pre-training autoencoder plays important role in our approach. You can find the pretrain model in repo as well.

zkharryhhhh commented 3 years ago

Thanks for your interest in our paper. As you already might notice, pre-training autoencoder plays important role in our approach. You can find the pretrain model in repo as well.

Thanks very much for @hadifar. And I'm sorry that I am late to reply. Your advice on pre-training autoencoder is correct. The pre-training autoencoder is important. I use your model for Stackoverflow and my own pre-trainning autoencoder for Stackoverflow which both get nice results just as your paper. But my problem of reproducing results is for another two datasets, Search Snippets and Biomedical Dataset. In your repo ,there are not pre-training autoencoder for above two datasets. So I use your model and data from xu2017(https://github.com/jacoxu/STC2/tree/master/dataset) just as your paper to get a pre-training autoencoder model. And then I get worse results what I described by using the pre-training model. Now I wonder if the hyperparameter is not proper or some other reasons. Can you give me some advice on the results for the other two datasets? If I express unclearly or the experiments have other settings , is there an email address so that I can contact you? And my email address is zhangkai2020c@iscas.ac.cn. I am trying some things for short text clustering based on your work, and look forward to communicate with you.