search

attardi / deepnl

Deep Learning for Natural Language Processing
GNU General Public License v3.0
457 stars 116 forks source link

Error Rate and Accuracy Sentiment Specific Embeddings #32

Closed maulikdang closed 8 years ago

maulikdang commented 8 years ago

I am using dl-sentiwords.py with the word2vec variant. (word2vec vector file downloaded from GloVe). The training runs successfully but the error increases with every epoch iteration and the accuracy remains at 0.00. Any inputs on what could be done to fix the issue?

maulik@maulik-VPCEH38FN:~/deepnl/bin$ python dl-sentiwords.py training.tsv --vectors vectors.txt --variant word2vec --vocab vocab.txt --model model1 -e 30 --hidden 20 Saving vocabulary in vocab.txt Creating new network... ... with the following parameters:

    Input layer size: 550
    Hidden layer size: 20
    Output size: 2

Starting training Epoch: 0, pairs: 10000, sent: 52656, avg. error: 4.176 1 epochs Examples: 10041 Error: 62460.347864 Accuracy: 0.000000 6925 corrections skipped Epoch: 1, pairs: 20000, sent: 52337, avg. error: 8.991 2 epochs Examples: 20082 Error: 207734.402282 Accuracy: 0.000000 7105 corrections skipped Epoch: 2, pairs: 30000, sent: 52020, avg. error: 14.748 3 epochs Examples: 30123 Error: 388989.344357 Accuracy: 0.000000 7052 corrections skipped Epoch: 3, pairs: 40000, sent: 51791, avg. error: 20.649 4 epochs Examples: 40164 Error: 573208.304635 Accuracy: 0.000000 6956 corrections skipped Epoch: 4, pairs: 50000, sent: 51598, avg. error: 26.750 5 epochs Examples: 50205 Error: 767234.504043 Accuracy: 0.000000 6906 corrections skipped Epoch: 5, pairs: 60000, sent: 51239, avg. error: 32.770 6 epochs Examples: 60246 Error: 951057.397979 Accuracy: 0.000000 6981 corrections skipped Epoch: 6, pairs: 70000, sent: 51166, avg. error: 38.861 7 epochs Examples: 70287 Error: 1129657.364728 Accuracy: 0.000000 7030 corrections skipped Epoch: 7, pairs: 80000, sent: 50936, avg. error: 45.031 8 epochs Examples: 80328 Error: 1316129.222171 Accuracy: 0.000000 6920 corrections skipped Epoch: 8, pairs: 90000, sent: 50857, avg. error: 51.284 9 epochs Examples: 90369 Error: 1530057.560660 Accuracy: 0.000000 6999 corrections skipped Epoch: 9, pairs: 100000, sent: 50718, avg. error: 57.600 10 epochs Examples: 100410 Error: 1705694.047717 Accuracy: 0.000000 7016 corrections skipped Epoch: 10, pairs: 110000, sent: 50425, avg. error: 63.825 11 epochs Examples: 110451 Error: 1882818.481616 Accuracy: 0.000000 7110 corrections skipped Epoch: 11, pairs: 120000, sent: 50216, avg. error: 69.894 12 epochs Examples: 120492 Error: 2074675.285961 Accuracy: 0.000000 7015 corrections skipped Epoch: 12, pairs: 130000, sent: 49730, avg. error: 76.021 13 epochs Examples: 130533 Error: 2265213.329245 Accuracy: 0.000000 7052 corrections skipped Epoch: 13, pairs: 140000, sent: 49498, avg. error: 82.320 14 epochs Examples: 140574 Error: 2463388.976950 Accuracy: 0.000000 7122 corrections skipped Epoch: 14, pairs: 150000, sent: 49306, avg. error: 88.331 15 epochs Examples: 150615 Error: 2622099.455269 Accuracy: 0.000000 7068 corrections skipped Epoch: 15, pairs: 160000, sent: 49051, avg. error: 94.802 16 epochs Examples: 160656 Error: 2853079.937031 Accuracy: 0.000000 7071 corrections skipped Epoch: 16, pairs: 170000, sent: 48561, avg. error: 101.018 17 epochs Examples: 170697 Error: 2995733.476832 Accuracy: 0.000000 7057 corrections skipped Epoch: 17, pairs: 180000, sent: 48292, avg. error: 107.027 18 epochs Examples: 180738 Error: 3206937.575615 Accuracy: 0.000000 6999 corrections skipped Epoch: 18, pairs: 190000, sent: 48281, avg. error: 113.053 19 epochs Examples: 190779 Error: 3404130.852323 Accuracy: 0.000000 7008 corrections skipped Epoch: 19, pairs: 200000, sent: 47593, avg. error: 119.447 20 epochs Examples: 200820 Error: 3593241.777677 Accuracy: 0.000000 7039 corrections skipped Epoch: 20, pairs: 210000, sent: 47402, avg. error: 126.057 21 epochs Examples: 210861 Error: 3837913.673458 Accuracy: 0.000000 7039 corrections skipped Epoch: 21, pairs: 220000, sent: 47181, avg. error: 132.405 22 epochs Examples: 220902 Error: 3961487.942201 Accuracy: 0.000000 7049 corrections skipped Epoch: 22, pairs: 230000, sent: 47081, avg. error: 138.319 23 epochs Examples: 230943 Error: 4089176.958714 Accuracy: 0.000000 7027 corrections skipped Epoch: 23, pairs: 240000, sent: 46938, avg. error: 144.327 24 epochs Examples: 240984 Error: 4287874.864616 Accuracy: 0.000000 7095 corrections skipped Epoch: 24, pairs: 250000, sent: 46900, avg. error: 150.805 25 epochs Examples: 251025 Error: 4578433.858292 Accuracy: 0.000000 7048 corrections skipped Epoch: 25, pairs: 260000, sent: 46743, avg. error: 156.881 26 epochs Examples: 261066 Error: 4756786.779839 Accuracy: 0.000000 6964 corrections skipped Epoch: 26, pairs: 270000, sent: 46546, avg. error: 162.995 27 epochs Examples: 271107 Error: 4879088.927508 Accuracy: 0.000000 6984 corrections skipped Epoch: 27, pairs: 280000, sent: 46313, avg. error: 169.356 28 epochs Examples: 281148 Error: 5061753.620850 Accuracy: 0.000000 7013 corrections skipped Epoch: 28, pairs: 290000, sent: 46215, avg. error: 175.712 29 epochs Examples: 291189 Error: 5285739.418424 Accuracy: 0.000000 7078 corrections skipped Epoch: 29, pairs: 300000, sent: 45871, avg. error: 181.846 30 epochs Examples: 301230 Error: 5465347.502520 Accuracy: 0.000000 7121 corrections skipped Overriding vectors to vectors.txt Saving trained model to model1

attardi commented 8 years ago

Don't worry about those numbers. You shoud get useable embeddings anyway.

On 29/4/2016 7:54, maulikdang wrote:

I am using dl-sentiwords.py with the word2vec variant. (word2vec vector file downloaded from GloVe).

The training runs successfully but the error increases with every
epoch iteration and the accuracy remains at 0.00. Any inputs on
what could be done to fix the issue?

maulik@maulik-VPCEH38FN:~/deepnl/bin$ python dl-sentiwords.py training.tsv --vectors vectors.txt --variant word2vec --vocab vocab.txt --model model1 -e 30 --hidden 20 Saving vocabulary in vocab.txt Creating new network... ... with the following parameters:

|Input layer size: 550 Hidden layer size: 20 Output size: 2 |

Starting training Epoch: 0, pairs: 10000, sent: 52656, avg. error: 4.176 1 epochs Examples: 10041 Error: 62460.347864 Accuracy: 0.000000 6925 corrections skipped Epoch: 1, pairs: 20000, sent: 52337, avg. error: 8.991 2 epochs Examples: 20082 Error: 207734.402282 Accuracy: 0.000000 7105 corrections skipped Epoch: 2, pairs: 30000, sent: 52020, avg. error: 14.748 3 epochs Examples: 30123 Error: 388989.344357 Accuracy: 0.000000 7052 corrections skipped Epoch: 3, pairs: 40000, sent: 51791, avg. error: 20.649 4 epochs Examples: 40164 Error: 573208.304635 Accuracy: 0.000000 6956 corrections skipped Epoch: 4, pairs: 50000, sent: 51598, avg. error: 26.750 5 epochs Examples: 50205 Error: 767234.504043 Accuracy: 0.000000 6906 corrections skipped Epoch: 5, pairs: 60000, sent: 51239, avg. error: 32.770 6 epochs Examples: 60246 Error: 951057.397979 Accuracy: 0.000000 6981 corrections skipped Epoch: 6, pairs: 70000, sent: 51166, avg. error: 38.861 7 epochs Examples: 70287 Error: 1129657.364728 Accuracy: 0.000000 7030 corrections skipped Epoch: 7, pairs: 80000, sent: 50936, avg. error: 45.031 8 epochs Examples: 80328 Error: 1316129.222171 Accuracy: 0.000000 6920 corrections skipped Epoch: 8, pairs: 90000, sent: 50857, avg. error: 51.284 9 epochs Examples: 90369 Error: 1530057.560660 Accuracy: 0.000000 6999 corrections skipped Epoch: 9, pairs: 100000, sent: 50718, avg. error: 57.600 10 epochs Examples: 100410 Error: 1705694.047717 Accuracy: 0.000000 7016 corrections skipped Epoch: 10, pairs: 110000, sent: 50425, avg. error: 63.825 11 epochs Examples: 110451 Error: 1882818.481616 Accuracy: 0.000000 7110 corrections skipped Epoch: 11, pairs: 120000, sent: 50216, avg. error: 69.894 12 epochs Examples: 120492 Error: 2074675.285961 Accuracy: 0.000000 7015 corrections skipped Epoch: 12, pairs: 130000, sent: 49730, avg. error: 76.021 13 epochs Examples: 130533 Error: 2265213.329245 Accuracy: 0.000000 7052 corrections skipped Epoch: 13, pairs: 140000, sent: 49498, avg. error: 82.320 14 epochs Examples: 140574 Error: 2463388.976950 Accuracy: 0.000000 7122 corrections skipped Epoch: 14, pairs: 150000, sent: 49306, avg. error: 88.331 15 epochs Examples: 150615 Error: 2622099.455269 Accuracy: 0.000000 7068 corrections skipped Epoch: 15, pairs: 160000, sent: 49051, avg. error: 94.802 16 epochs Examples: 160656 Error: 2853079.937031 Accuracy: 0.000000 7071 corrections skipped Epoch: 16, pairs: 170000, sent: 48561, avg. error: 101.018 17 epochs Examples: 170697 Error: 2995733.476832 Accuracy: 0.000000 7057 corrections skipped Epoch: 17, pairs: 180000, sent: 48292, avg. error: 107.027 18 epochs Examples: 180738 Error: 3206937.575615 Accuracy: 0.000000 6999 corrections skipped Epoch: 18, pairs: 190000, sent: 48281, avg. error: 113.053 19 epochs Examples: 190779 Error: 3404130.852323 Accuracy: 0.000000 7008 corrections skipped Epoch: 19, pairs: 200000, sent: 47593, avg. error: 119.447 20 epochs Examples: 200820 Error: 3593241.777677 Accuracy: 0.000000 7039 corrections skipped Epoch: 20, pairs: 210000, sent: 47402, avg. error: 126.057 21 epochs Examples: 210861 Error: 3837913.673458 Accuracy: 0.000000 7039 corrections skipped Epoch: 21, pairs: 220000, sent: 47181, avg. error: 132.405 22 epochs Examples: 220902 Error: 3961487.942201 Accuracy: 0.000000 7049 corrections skipped Epoch: 22, pairs: 230000, sent: 47081, avg. error: 138.319 23 epochs Examples: 230943 Error: 4089176.958714 Accuracy: 0.000000 7027 corrections skipped Epoch: 23, pairs: 240000, sent: 46938, avg. error: 144.327 24 epochs Examples: 240984 Error: 4287874.864616 Accuracy: 0.000000 7095 corrections skipped Epoch: 24, pairs: 250000, sent: 46900, avg. error: 150.805 25 epochs Examples: 251025 Error: 4578433.858292 Accuracy: 0.000000 7048 corrections skipped Epoch: 25, pairs: 260000, sent: 46743, avg. error: 156.881 26 epochs Examples: 261066 Error: 4756786.779839 Accuracy: 0.000000 6964 corrections skipped Epoch: 26, pairs: 270000, sent: 46546, avg. error: 162.995 27 epochs Examples: 271107 Error: 4879088.927508 Accuracy: 0.000000 6984 corrections skipped Epoch: 27, pairs: 280000, sent: 46313, avg. error: 169.356 28 epochs Examples: 281148 Error: 5061753.620850 Accuracy: 0.000000 7013 corrections skipped Epoch: 28, pairs: 290000, sent: 46215, avg. error: 175.712 29 epochs Examples: 291189 Error: 5285739.418424 Accuracy: 0.000000 7078 corrections skipped Epoch: 29, pairs: 300000, sent: 45871, avg. error: 181.846 30 epochs Examples: 301230 Error: 5465347.502520 Accuracy: 0.000000 7121 corrections skipped Overriding vectors to vectors.txt Saving trained model to model1

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/attardi/deepnl/issues/32

maulikdang commented 8 years ago

okay sir, thank you. That means after the training i will get sentiment specific embeddings(despite the error rate) that can be used directly?

attardi commented 8 years ago

Correct.