search

dinghanshen / SWEM

The Tensorflow code for this ACL 2018 paper: "Baseline Needs More Love: On Simple Word-Embedding-Based Models and Associated Pooling Mechanisms"
283 stars 54 forks source link

Why simple average pooling of w2v should be better than LSTM and some CNNs? Unbelievable! #9

Open beyondguo opened 5 years ago

beyondguo commented 5 years ago

read through the paper, I didn’t find what w2v embedding other models(such as LSTM,CNN) are using. It is amazing that SWEM -ave can achieve better results than LSTM or CNN in some tasks, which in fact I don’t believe! I have done a lot of nlp tasks and I know that simply average the word embedding of a text is usually very poor. I don’t think the comparisons of other models are fair. They don’t even use the same pretrained w2v. So maybe it’s just the Glove you used is better than the embedding other models used.

dinghanshen commented 5 years ago

To your questions/concerns:

First, the same word embeddings (Glove) were used for both LSTM and CNN to ensure fair comparison.

Second, you should be really careful while implementing the simple average operation to obtain sentence embeddings. Specifically, a mask is needed to remove the influence of padding tokens. Otherwise the paddings will hurt the final performance a lot. This is what I have implemented in this codebase.

Finally, we have tried to shuffle the words in every sentence and the LSTM gives the same accuracy. In other words, LSTM does not leverage the word order information at all, or the information is not useful for the tasks considered.

beyondguo commented 5 years ago

Thanks for replying so fast. In paper [Character-level Convolutional Networks for TextClassification] where the results of LSTM/bag-of-means/CNNs are reported, I couldn't find the evidence that they were using Glove. Actually, that paper didn't even cite Pennington's Glove paper. I only noticed that both of you used the same dimension of w2v---300. So I still have doubts for this.

As for the trick for average pooling that mask should be applied, I also noticed it from your code. I will try it and see its effectiveness. Thanks for this important information.

Although I still have soncerns,I believe all the results in your paper are true. So if the simple baseline can beat so many powerful complicated models such as LSTM and CNNs, dose that mean the LSTM and CNNs and fasttext have no use, which is widely implemented in industry and reasearch? Because even they are slightly better than SWEM, their computational cost is much higher.

Cppowboy commented 4 years ago

Is the LSTM or CNN model implemented with mask?