search

arjunmenon / smalltext

Classify short texts with neural network.
MIT License
16 stars 1 forks source link

Issue with larger sizes #3

Closed wflanagan closed 4 years ago

wflanagan commented 4 years ago

Hey,

I tried to use your gem to do a model build against classifier-reborn and a few others. I used a rather large data set (more data is better, right?) of 7.1 million entries. It failed.. did not provide any input for more than 24 hours.. and in general, didn't really work.

Thought I'd make you aware, and I'm happy to supply the dataset if you're interested.

Wf

arjunmenon commented 4 years ago

Hi This gem specifically serves scenarios where dataset is sparse. Many times you want to build a model with a small dataset because you don't have much data and just begin classification. It is not designed to handle large dataset. For that I would recommend using Facebook's FastText.