search

Tencent / NeuralNLP-NeuralClassifier

An Open-source Neural Hierarchical Multi-label Text Classification Toolkit
Other
1.83k stars 402 forks source link

Reproducing the results on RCV1 #56

Closed soumyac1999 closed 4 years ago

soumyac1999 commented 4 years ago

I've tried everything that was mentioned in reply to various issues but I am still not able to reproduce the results given in the README.

The results that I have got are: -- Micro F1 Macro F1
Flat 0.8354 0.5801
Hierarchical 0.8360 0.5798

Can you share the json configuration files used for getting the values you have reported?

ayushbits commented 4 years ago

@liqunhit @coderbyr Even I am not able to reproduce the reported numbers in Table 3 using RCNN for both Hierarchical and Flat. The numbers i've received is similar to what is posted above.

RCV1 dataset used is the original dataset from trec and glove pretrained embeddings of 300 dim. Could you see what could be the issue ?

Thanks

coderbyr commented 4 years ago

maybe two reasons : 1、actually we use 10% of the origin train as the dev set, and using the other as the train set, it may get different results 2、we did't try all groups of the hyper parameters, there could be better results than what we got in the paper

soumyac1999 commented 4 years ago

@coderbyr

  1. I am pretty sure that the choice of the 10% dev set should not improve the results of the flat model from (0.7955, 0.5123) to (0.8354, 0.5801). Still if you think that is the case, do you mind sharing the code which you used to generate the said split from raw RCV1 dataset?
  2. My concern is not getting better results; all I want to know is the set of hyperparameters which you have used to get the reported results on Flat RCV1 with TextCNN. I haven't changed the hyperparameters from what is given in conf.json. Can you please share the config file which you have used to get the reported results?