fasttext loss and the long text classification

brightmart / text_classification

all kinds of text classification models and more with deep learning

MIT License

7.83k stars 2.57k forks source link

Closed songzenghui closed 6 years ago

songzenghui commented 6 years ago

NICE project! I have two issues,could u or someone else give me some idea?

Why the fasttext 's loss is different between training and eval?
How to deal with the long text classification ,which its max length maybe 40,000 min length maybe 10,but its most length is 500-600 Thank you

brightmart commented 6 years ago

hi,

basically, use a special loss for training to speed up the training process.

for long text classification, you can check models under this repository, which is handling documents, and the average length is around 400: