Open timothywangdev opened 9 years ago
Collection of experiment results
RC_2015-03 (32.6 GB uncompressed)
ubuntu@ip-172-31-6-ive 5 -size 300 -output output -train ../data_generated -min-count 50 -threads 16 -iter 10 -cbow 0
Starting training using file ../data_generated
Vocab size: 203291
Words in train file: 1830337168
Alpha: 0.000002 Progress: 100.00% Words/thread/sec: 66.75k
ubuntu@ip-172-31-6ords.txtord2vec$ ./compute-accuracy output 30000 < questions-w
capital-common-countries:
ACCURACY TOP1: 46.15 % (84 / 182)
Total accuracy: 46.15 % Semantic accuracy: 46.15 % Syntactic accuracy: -nan %
capital-world:
ACCURACY TOP1: 46.37 % (83 / 179)
Total accuracy: 46.26 % Semantic accuracy: 46.26 % Syntactic accuracy: -nan %
currency:
ACCURACY TOP1: 3.70 % (2 / 54)
Total accuracy: 40.72 % Semantic accuracy: 40.72 % Syntactic accuracy: -nan %
city-in-state:
ACCURACY TOP1: 13.47 % (139 / 1032)
Total accuracy: 21.29 % Semantic accuracy: 21.29 % Syntactic accuracy: -nan %
family:
ACCURACY TOP1: 92.81 % (284 / 306)
Total accuracy: 33.77 % Semantic accuracy: 33.77 % Syntactic accuracy: -nan %
gram1-adjective-to-adverb:
ACCURACY TOP1: 23.01 % (214 / 930)
Total accuracy: 30.04 % Semantic accuracy: 33.77 % Syntactic accuracy: 23.01 %
gram2-opposite:
ACCURACY TOP1: 53.83 % (323 / 600)
Total accuracy: 34.39 % Semantic accuracy: 33.77 % Syntactic accuracy: 35.10 %
gram3-comparative:
ACCURACY TOP1: 91.07 % (1213 / 1332)
Total accuracy: 50.75 % Semantic accuracy: 33.77 % Syntactic accuracy: 61.15 %
gram4-superlative:
ACCURACY TOP1: 91.14 % (689 / 756)
Total accuracy: 56.43 % Semantic accuracy: 33.77 % Syntactic accuracy: 67.41 %
gram5-present-participle:
ACCURACY TOP1: 77.84 % (822 / 1056)
Total accuracy: 59.95 % Semantic accuracy: 33.77 % Syntactic accuracy: 69.77 %
gram6-nationality-adjective:
ACCURACY TOP1: 75.47 % (597 / 791)
Total accuracy: 61.65 % Semantic accuracy: 33.77 % Syntactic accuracy: 70.59 %
gram7-past-tense:
ACCURACY TOP1: 62.42 % (925 / 1482)
Total accuracy: 61.78 % Semantic accuracy: 33.77 % Syntactic accuracy: 68.85 %
gram8-plural:
ACCURACY TOP1: 82.20 % (868 / 1056)
Total accuracy: 63.99 % Semantic accuracy: 33.77 % Syntactic accuracy: 70.61 %
gram9-plural-verbs:
ACCURACY TOP1: 83.76 % (588 / 702)
Total accuracy: 65.32 % Semantic accuracy: 33.77 % Syntactic accuracy: 71.67 %
Questions seen / total: 10458 19544 53.51 %
We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)
What machines are you using to train? My laptop isn't really cutting it. Like 0.5%/hr ...
i7 4800 MQ or ec2 c4.4xlarge
RC_2015-01 (31G uncompressed)
But corpus was generated without any splitting of comments, so each comment's body was fully on one line.
time ./word2vec -train data_generated -output vectors.bin -cbow 1 -size 200 -window 5 -negative 5 -hs 0 -threads 4 -binary 1 -iter 5 -min-count 50
osboxes@osboxes:~/Desktop/word2vec/trunk$ ./compute-accuracy vectors.bin 30000 < questions-words.txt
capital-common-countries:
ACCURACY TOP1: 42.86 % (90 / 210)
Total accuracy: 42.86 % Semantic accuracy: 42.86 % Syntactic accuracy: -nan %
capital-world:
ACCURACY TOP1: 39.29 % (77 / 196)
Total accuracy: 41.13 % Semantic accuracy: 41.13 % Syntactic accuracy: -nan %
currency:
ACCURACY TOP1: 8.57 % (6 / 70)
Total accuracy: 36.34 % Semantic accuracy: 36.34 % Syntactic accuracy: -nan %
city-in-state:
ACCURACY TOP1: 9.92 % (84 / 847)
Total accuracy: 19.43 % Semantic accuracy: 19.43 % Syntactic accuracy: -nan %
family:
ACCURACY TOP1: 88.89 % (272 / 306)
Total accuracy: 32.47 % Semantic accuracy: 32.47 % Syntactic accuracy: -nan %
gram1-adjective-to-adverb:
ACCURACY TOP1: 17.85 % (166 / 930)
Total accuracy: 27.16 % Semantic accuracy: 32.47 % Syntactic accuracy: 17.85 %
gram2-opposite:
ACCURACY TOP1: 44.33 % (266 / 600)
Total accuracy: 30.42 % Semantic accuracy: 32.47 % Syntactic accuracy: 28.24 %
gram3-comparative:
ACCURACY TOP1: 87.69 % (1168 / 1332)
Total accuracy: 47.41 % Semantic accuracy: 32.47 % Syntactic accuracy: 55.90 %
gram4-superlative:
ACCURACY TOP1: 81.18 % (755 / 930)
Total accuracy: 53.20 % Semantic accuracy: 32.47 % Syntactic accuracy: 62.10 %
gram5-present-participle:
ACCURACY TOP1: 82.29 % (869 / 1056)
Total accuracy: 57.94 % Semantic accuracy: 32.47 % Syntactic accuracy: 66.50 %
gram6-nationality-adjective:
ACCURACY TOP1: 52.83 % (448 / 848)
Total accuracy: 57.35 % Semantic accuracy: 32.47 % Syntactic accuracy: 64.47 %
gram7-past-tense:
ACCURACY TOP1: 66.53 % (986 / 1482)
Total accuracy: 58.90 % Semantic accuracy: 32.47 % Syntactic accuracy: 64.89 %
gram8-plural:
ACCURACY TOP1: 75.47 % (797 / 1056)
Total accuracy: 60.67 % Semantic accuracy: 32.47 % Syntactic accuracy: 66.25 %
gram9-plural-verbs:
ACCURACY TOP1: 80.77 % (567 / 702)
Total accuracy: 62.01 % Semantic accuracy: 32.47 % Syntactic accuracy: 67.39 %
Questions seen / total: 10565 19544 54.06 %
looks like it's got a higher semantic accuracy, not sure if it's due to a larger word vector dimension. I'm training a much larger dataset right now (entire 2015 dataset), hopefully we will get a better semantic accuracy.
1tb on ec2? isnt the storage alone pretty expensive?
RC_2015-01,02,03,04,05
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./word2vec -binary 1 -save-vocab vocab -negative 3 -size 800 -output output -train ../ConversationalAgent/experiments/word2vec/data/data_generated -min-count 100 -threads 8 -iter 5
Starting training using file ../ConversationalAgent/experiments/word2vec/data/data_generated
Vocab size: 360272
Words in train file: 8921611451
Alpha: 0.000036 Progress: 99.93% Words/thread/sec: 119.62k
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./compute-accuracy output 30000 < questions-words.txt
capital-common-countries:
ACCURACY TOP1: 45.60 % (83 / 182)
Total accuracy: 45.60 % Semantic accuracy: 45.60 % Syntactic accuracy: -nan %
capital-world:
ACCURACY TOP1: 39.66 % (71 / 179)
Total accuracy: 42.66 % Semantic accuracy: 42.66 % Syntactic accuracy: -nan %
currency:
ACCURACY TOP1: 0.00 % (0 / 54)
Total accuracy: 37.11 % Semantic accuracy: 37.11 % Syntactic accuracy: -nan %
city-in-state:
ACCURACY TOP1: 17.55 % (156 / 889)
Total accuracy: 23.77 % Semantic accuracy: 23.77 % Syntactic accuracy: -nan %
family:
ACCURACY TOP1: 97.06 % (297 / 306)
Total accuracy: 37.70 % Semantic accuracy: 37.70 % Syntactic accuracy: -nan %
gram1-adjective-to-adverb:
ACCURACY TOP1: 21.40 % (199 / 930)
Total accuracy: 31.73 % Semantic accuracy: 37.70 % Syntactic accuracy: 21.40 %
gram2-opposite:
ACCURACY TOP1: 49.83 % (299 / 600)
Total accuracy: 35.19 % Semantic accuracy: 37.70 % Syntactic accuracy: 32.55 %
gram3-comparative:
ACCURACY TOP1: 91.22 % (1215 / 1332)
Total accuracy: 51.88 % Semantic accuracy: 37.70 % Syntactic accuracy: 59.85 %
gram4-superlative:
ACCURACY TOP1: 85.85 % (649 / 756)
Total accuracy: 56.79 % Semantic accuracy: 37.70 % Syntactic accuracy: 65.28 %
gram5-present-participle:
ACCURACY TOP1: 84.09 % (888 / 1056)
Total accuracy: 61.38 % Semantic accuracy: 37.70 % Syntactic accuracy: 69.53 %
gram6-nationality-adjective:
ACCURACY TOP1: 65.11 % (515 / 791)
Total accuracy: 61.80 % Semantic accuracy: 37.70 % Syntactic accuracy: 68.89 %
gram7-past-tense:
ACCURACY TOP1: 71.93 % (1066 / 1482)
Total accuracy: 63.55 % Semantic accuracy: 37.70 % Syntactic accuracy: 69.54 %
gram8-plural:
ACCURACY TOP1: 87.78 % (927 / 1056)
Total accuracy: 66.21 % Semantic accuracy: 37.70 % Syntactic accuracy: 71.95 %
gram9-plural-verbs:
ACCURACY TOP1: 80.34 % (564 / 702)
Total accuracy: 67.17 % Semantic accuracy: 37.70 % Syntactic accuracy: 72.62 %
Questions seen / total: 10315 19544 52.78 %
Not a significant improvement
Collection of experiment results
We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)