search

timothywangdev / ConversationalAgent

Conversational Agent
1 stars 0 forks source link

word2vec accuracy #10

Open timothywangdev opened 9 years ago

timothywangdev commented 9 years ago

Collection of experiment results

RC_2015-03 (32.6 GB uncompressed)
hujie@hujie-GT70-2PE:~/projects/ml/word2vec$ ./word2vec -binary 1 -save-vocab vocab -negative 5 -size 500 -output output -train ../ConversationalAgent/experiments/word2vec/data/data_generated -min-count 50 -threads 8
Starting training using file ../ConversationalAgent/experiments/word2vec/data/data_generated
Vocab size: 203291
Words in train file: 1830337168
Alpha: 0.000005  Progress: 100.00%  Words/thread/sec: 131.63k
hujie@hujie-GT70-2PE:~/projects/ml/word2vec$ ./compute-accuracy output 30000 < questions-words.txt
capital-common-countries:
ACCURACY TOP1: 30.77 %  (56 / 182)
Total accuracy: 30.77 %   Semantic accuracy: 30.77 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 22.35 %  (40 / 179)
Total accuracy: 26.59 %   Semantic accuracy: 26.59 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 1.85 %  (1 / 54)
Total accuracy: 23.37 %   Semantic accuracy: 23.37 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 8.72 %  (90 / 1032)
Total accuracy: 12.92 %   Semantic accuracy: 12.92 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 94.12 %  (288 / 306)
Total accuracy: 27.10 %   Semantic accuracy: 27.10 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 16.77 %  (156 / 930)
Total accuracy: 23.52 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 16.77 % 
gram2-opposite:
ACCURACY TOP1: 46.50 %  (279 / 600)
Total accuracy: 27.72 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 28.43 % 
gram3-comparative:
ACCURACY TOP1: 89.71 %  (1195 / 1332)
Total accuracy: 45.61 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 56.95 % 
gram4-superlative:
ACCURACY TOP1: 77.25 %  (584 / 756)
Total accuracy: 50.07 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 61.19 % 
gram5-present-participle:
ACCURACY TOP1: 79.83 %  (843 / 1056)
Total accuracy: 54.96 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 65.40 % 
gram6-nationality-adjective:
ACCURACY TOP1: 56.38 %  (446 / 791)
Total accuracy: 55.11 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 64.10 % 
gram7-past-tense:
ACCURACY TOP1: 69.16 %  (1025 / 1482)
Total accuracy: 57.51 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 65.18 % 
gram8-plural:
ACCURACY TOP1: 83.33 %  (880 / 1056)
Total accuracy: 60.30 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 67.57 % 
gram9-plural-verbs:
ACCURACY TOP1: 77.78 %  (546 / 702)
Total accuracy: 61.47 %   Semantic accuracy: 27.10 %   Syntactic accuracy: 68.40 % 
Questions seen / total: 10458 19544   53.51 %

We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)

timothywangdev commented 9 years ago

Collection of experiment results

RC_2015-03 (32.6 GB uncompressed)
ubuntu@ip-172-31-6-ive 5 -size 300 -output output -train ../data_generated -min-count 50 -threads 16 -iter 10 -cbow 0
Starting training using file ../data_generated
Vocab size: 203291
Words in train file: 1830337168
Alpha: 0.000002  Progress: 100.00%  Words/thread/sec: 66.75k 
ubuntu@ip-172-31-6ords.txtord2vec$ ./compute-accuracy output 30000 < questions-w 
capital-common-countries:
ACCURACY TOP1: 46.15 %  (84 / 182)
Total accuracy: 46.15 %   Semantic accuracy: 46.15 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 46.37 %  (83 / 179)
Total accuracy: 46.26 %   Semantic accuracy: 46.26 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 3.70 %  (2 / 54)
Total accuracy: 40.72 %   Semantic accuracy: 40.72 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 13.47 %  (139 / 1032)
Total accuracy: 21.29 %   Semantic accuracy: 21.29 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 92.81 %  (284 / 306)
Total accuracy: 33.77 %   Semantic accuracy: 33.77 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 23.01 %  (214 / 930)
Total accuracy: 30.04 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 23.01 % 
gram2-opposite:
ACCURACY TOP1: 53.83 %  (323 / 600)
Total accuracy: 34.39 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 35.10 % 
gram3-comparative:
ACCURACY TOP1: 91.07 %  (1213 / 1332)
Total accuracy: 50.75 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 61.15 % 
gram4-superlative:
ACCURACY TOP1: 91.14 %  (689 / 756)
Total accuracy: 56.43 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 67.41 % 
gram5-present-participle:
ACCURACY TOP1: 77.84 %  (822 / 1056)
Total accuracy: 59.95 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 69.77 % 
gram6-nationality-adjective:
ACCURACY TOP1: 75.47 %  (597 / 791)
Total accuracy: 61.65 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 70.59 % 
gram7-past-tense:
ACCURACY TOP1: 62.42 %  (925 / 1482)
Total accuracy: 61.78 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 68.85 % 
gram8-plural:
ACCURACY TOP1: 82.20 %  (868 / 1056)
Total accuracy: 63.99 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 70.61 % 
gram9-plural-verbs:
ACCURACY TOP1: 83.76 %  (588 / 702)
Total accuracy: 65.32 %   Semantic accuracy: 33.77 %   Syntactic accuracy: 71.67 % 
Questions seen / total: 10458 19544   53.51 %

We got low accuracy in several tests(currency, city-in-state, capital-world, capital-common-countries)

bwuu commented 9 years ago

What machines are you using to train? My laptop isn't really cutting it. Like 0.5%/hr ...

timothywangdev commented 9 years ago

i7 4800 MQ or ec2 c4.4xlarge

bwuu commented 9 years ago 
RC_2015-01  (31G uncompressed)

But corpus was generated without any splitting of comments, so each comment's body was fully on one line.

time ./word2vec -train data_generated -output vectors.bin -cbow 1 -size 200 -window 5 -negative 5 -hs 0 -threads 4 -binary 1 -iter 5 -min-count 50
osboxes@osboxes:~/Desktop/word2vec/trunk$ ./compute-accuracy vectors.bin 30000 < questions-words.txt 
capital-common-countries:
ACCURACY TOP1: 42.86 %  (90 / 210)
Total accuracy: 42.86 %   Semantic accuracy: 42.86 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 39.29 %  (77 / 196)
Total accuracy: 41.13 %   Semantic accuracy: 41.13 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 8.57 %  (6 / 70)
Total accuracy: 36.34 %   Semantic accuracy: 36.34 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 9.92 %  (84 / 847)
Total accuracy: 19.43 %   Semantic accuracy: 19.43 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 88.89 %  (272 / 306)
Total accuracy: 32.47 %   Semantic accuracy: 32.47 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 17.85 %  (166 / 930)
Total accuracy: 27.16 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 17.85 % 
gram2-opposite:
ACCURACY TOP1: 44.33 %  (266 / 600)
Total accuracy: 30.42 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 28.24 % 
gram3-comparative:
ACCURACY TOP1: 87.69 %  (1168 / 1332)
Total accuracy: 47.41 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 55.90 % 
gram4-superlative:
ACCURACY TOP1: 81.18 %  (755 / 930)
Total accuracy: 53.20 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 62.10 % 
gram5-present-participle:
ACCURACY TOP1: 82.29 %  (869 / 1056)
Total accuracy: 57.94 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 66.50 % 
gram6-nationality-adjective:
ACCURACY TOP1: 52.83 %  (448 / 848)
Total accuracy: 57.35 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 64.47 % 
gram7-past-tense:
ACCURACY TOP1: 66.53 %  (986 / 1482)
Total accuracy: 58.90 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 64.89 % 
gram8-plural:
ACCURACY TOP1: 75.47 %  (797 / 1056)
Total accuracy: 60.67 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 66.25 % 
gram9-plural-verbs:
ACCURACY TOP1: 80.77 %  (567 / 702)
Total accuracy: 62.01 %   Semantic accuracy: 32.47 %   Syntactic accuracy: 67.39 % 
Questions seen / total: 10565 19544   54.06 %
timothywangdev commented 9 years ago

looks like it's got a higher semantic accuracy, not sure if it's due to a larger word vector dimension. I'm training a much larger dataset right now (entire 2015 dataset), hopefully we will get a better semantic accuracy.

bwuu commented 9 years ago

1tb on ec2? isnt the storage alone pretty expensive?

timothywangdev commented 9 years ago 
RC_2015-01,02,03,04,05
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./word2vec -binary 1 -save-vocab vocab -negative 3 -size 800 -output output -train ../ConversationalAgent/experiments/word2vec/data/data_generated -min-count 100 -threads 8 -iter 5
Starting training using file ../ConversationalAgent/experiments/word2vec/data/data_generated
Vocab size: 360272
Words in train file: 8921611451
Alpha: 0.000036  Progress: 99.93%  Words/thread/sec: 119.62k
hehe@hehe-Q400A:~/projects/ml/word2vec$ ./compute-accuracy output 30000 < questions-words.txt 
capital-common-countries:
ACCURACY TOP1: 45.60 %  (83 / 182)
Total accuracy: 45.60 %   Semantic accuracy: 45.60 %   Syntactic accuracy: -nan % 
capital-world:
ACCURACY TOP1: 39.66 %  (71 / 179)
Total accuracy: 42.66 %   Semantic accuracy: 42.66 %   Syntactic accuracy: -nan % 
currency:
ACCURACY TOP1: 0.00 %  (0 / 54)
Total accuracy: 37.11 %   Semantic accuracy: 37.11 %   Syntactic accuracy: -nan % 
city-in-state:
ACCURACY TOP1: 17.55 %  (156 / 889)
Total accuracy: 23.77 %   Semantic accuracy: 23.77 %   Syntactic accuracy: -nan % 
family:
ACCURACY TOP1: 97.06 %  (297 / 306)
Total accuracy: 37.70 %   Semantic accuracy: 37.70 %   Syntactic accuracy: -nan % 
gram1-adjective-to-adverb:
ACCURACY TOP1: 21.40 %  (199 / 930)
Total accuracy: 31.73 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 21.40 % 
gram2-opposite:
ACCURACY TOP1: 49.83 %  (299 / 600)
Total accuracy: 35.19 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 32.55 % 
gram3-comparative:
ACCURACY TOP1: 91.22 %  (1215 / 1332)
Total accuracy: 51.88 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 59.85 % 
gram4-superlative:
ACCURACY TOP1: 85.85 %  (649 / 756)
Total accuracy: 56.79 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 65.28 % 
gram5-present-participle:
ACCURACY TOP1: 84.09 %  (888 / 1056)
Total accuracy: 61.38 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 69.53 % 
gram6-nationality-adjective:
ACCURACY TOP1: 65.11 %  (515 / 791)
Total accuracy: 61.80 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 68.89 % 
gram7-past-tense:
ACCURACY TOP1: 71.93 %  (1066 / 1482)
Total accuracy: 63.55 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 69.54 % 
gram8-plural:
ACCURACY TOP1: 87.78 %  (927 / 1056)
Total accuracy: 66.21 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 71.95 % 
gram9-plural-verbs:
ACCURACY TOP1: 80.34 %  (564 / 702)
Total accuracy: 67.17 %   Semantic accuracy: 37.70 %   Syntactic accuracy: 72.62 % 
Questions seen / total: 10315 19544   52.78 %

Not a significant improvement