DSKSD / DeepNLP-models-Pytorch

Pytorch implementations of various Deep NLP models in cs-224n(Stanford Univ)
MIT License
2.95k stars 660 forks source link

Input sequences contain part of the outputs in CNN-for-Text-Classification #8

Open ducalpha opened 6 years ago

ducalpha commented 6 years ago

Step [54] "data = [[d.split(':')[1][:-1], d.split(':')[0]] for d in data]" seems to include the sub-category output into the input sequence. For example, for data line "DESC:def What is ethology ?", data will be ["def What is ethology ?", "DESC"] so the "def" sub-category is included into the input sequence.

I suggest a fix:

# Remove the sub-category (first word) and the '?' at the end.
data = [[d.split(':')[1].split(' ', 1)[1][:-2], d.split(':')[0]] for d in data]