Implement kNN for tf.idf similarity and use it for prediction

soton-data-mining / job-salary-prediction

A regression problem, predicting salaries of jobs in UK based on various criteria

8 stars 3 forks source link

Implement kNN for tf.idf similarity and use it for prediction #32

Closed blanche closed 7 years ago

blanche commented 7 years ago

also

separate vectorization from similarity/knn computation
use stopwords & lowercase in vectorizer
use the same random seed for every model (still random, but better for comparison)
output model name when printing error

utkuozbulak commented 7 years ago

Can you please not merge this just for a little more while, I we, I will check it as soon as possible.

blanche commented 7 years ago

sure! no rush for this one.. my next task is cleaning up the companies anyway

btw, the knn thing its pretty fast, even on 100k data set (i.e. < 5 min?), but i get memory error on my machine when running it on the full 120k, maybe have to play with the parallelization..

blanche commented 7 years ago

done error = 7.4k