search

johnb30 / py_crepe

Keras implementation of the Crepe character-level convolutional neural net.
74 stars 33 forks source link

issue about accuracy on test set #5

Closed rezazad68 closed 7 years ago

rezazad68 commented 7 years ago

hello, at first thank for your implementation

I just executed code for 10 epochs (default setting on main.py) and the accuracy that I get on test set was 25%, it predicts completely random not the accuracy on paper!

which part of code I should change for reproducing paper result?

it would be great to explain the reason

with regards

johnb30 commented 7 years ago

A few things might be wrong.

1) What version of keras are you using? 2) You probably need to run it for more than 10 epochs. I think the full model takes a couple hundred epochs to run. 3) Output generated by the program would be useful to see.

rezazad68 commented 7 years ago

hey john

just solved the problem, Keras is using uniform weight as initial weights for all the layers, so I just changed it to normal distribution with mean zero and 0.05 variance and it worked well. its converge to the correct accuracy in about 200 epochs.

btw thank for your response

with regards

On Tue, Mar 7, 2017 at 6:56 AM, John Beieler notifications@github.com wrote:

A few things might be wrong.

  1. What version of keras are you using?
  2. You probably need to run it for more than 10 epochs. I think the full model takes a couple hundred epochs to run.
  3. Output generated by the program would be useful to see.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/johnb30/py_crepe/issues/5#issuecomment-284610718, or mute the thread https://github.com/notifications/unsubscribe-auth/ATnx4SbfboNyabVjIeBx1SMFKytZ-aidks5rjM5QgaJpZM4MP4z2 .

ayrtondenner commented 6 years ago

As I've stated in https://github.com/johnb30/py_crepe/issues/8#issuecomment-399919147, I runned the code "as is", without adding RandomNormal, and it managed to get 87%~92% of accuracy.

liufuyang commented 6 years ago

@ayrtondenner Did you also use tensorflow with GPU as backend? The code didn't work for me with a newer version of Keras2.1.5 + tf 1.6 gpu backend. But I didn't try it recently, so not sure if some other version updates will make the code work again.

liufuyang commented 6 years ago

@ayrtondenner Also looking at your log, you should notice that those 92% acc are actually evaluated on training set. So your actual test accuracy is at 87%.

ayrtondenner commented 6 years ago

Yes, indeed, I didn't pay attention to that. We both got 87% of accuracy in test/validation, so it seems.

ayrtondenner commented 6 years ago

After running the original code through the cmd, using THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python main.py, Python 3.6.5, Keras 2.1.5, and with TensorFlow 1.7.0 as backend, I got the following results:

Epoch: 8
  Step: 100
        Loss: 0.24006174571812153. Accuracy: 0.9212500113248825
  Step: 200
        Loss: 0.24197643373161554. Accuracy: 0.9206250131130218
  Step: 300
        Loss: 0.24339371661345163. Accuracy: 0.9196250134706497
  Step: 400
        Loss: 0.2464142149500549. Accuracy: 0.9168750141561032
  Step: 500
        Loss: 0.24290501976013185. Accuracy: 0.9180000140666962
  Step: 600
        Loss: 0.24499551185717186. Accuracy: 0.9170625145236652
  Step: 700
        Loss: 0.24328476248042924. Accuracy: 0.9176785854782377
  Step: 800
        Loss: 0.24540792437270284. Accuracy: 0.9165625140815973
  Step: 900
        Loss: 0.24504088713890976. Accuracy: 0.9166389031542672
  Step: 1000
        Loss: 0.24667447752505542. Accuracy: 0.9160375146269798
  Step: 1100
        Loss: 0.2472187682783062. Accuracy: 0.9158863780715248
  Step: 1200
        Loss: 0.24948887835567196. Accuracy: 0.9148645987113316
  Step: 1300
        Loss: 0.2499590342778426. Accuracy: 0.9145961692241522
  Step: 1400
        Loss: 0.2490278405270406. Accuracy: 0.9150535864063671
  Step: 1500
        Loss: 0.25050772219896317. Accuracy: 0.91466668232282
Epoch 8. Loss: 0.33384477543203456. Accuracy: 0.8817105569337543
Epoch time: 0:03:30.092366. Total time: 0:31:39.657972

Epoch: 9
  Step: 100
        Loss: 0.22639427155256273. Accuracy: 0.9212500095367432
  Step: 200
        Loss: 0.22508799128234386. Accuracy: 0.9226250100135803
  Step: 300
        Loss: 0.22923665245374045. Accuracy: 0.9203333441416423
  Step: 400
        Loss: 0.22523623278364538. Accuracy: 0.9227187602221966
  Step: 500
        Loss: 0.2268564992696047. Accuracy: 0.9225500104427338
  Step: 600
        Loss: 0.22460318664709727. Accuracy: 0.9233958429098129
  Step: 700
        Loss: 0.2252830421924591. Accuracy: 0.9235357241971153
  Step: 800
        Loss: 0.2271577899903059. Accuracy: 0.9222812597453595
  Step: 900
        Loss: 0.22651672060290973. Accuracy: 0.9225694541798698
  Step: 1000
        Loss: 0.22688820595294237. Accuracy: 0.922275009572506
  Step: 1100
        Loss: 0.22888841693374243. Accuracy: 0.9216704646023837
  Step: 1200
        Loss: 0.23034910179674625. Accuracy: 0.9209166769186655
  Step: 1300
        Loss: 0.2308478689365662. Accuracy: 0.9208942410120597
  Step: 1400
        Loss: 0.2306387787418706. Accuracy: 0.9209642959918295
  Step: 1500
        Loss: 0.23129621476928394. Accuracy: 0.9208333437045415
Epoch 9. Loss: 0.37526395995365947. Accuracy: 0.8742105571847213
Epoch time: 0:03:29.562941. Total time: 0:35:10.413084

It seems that the test accuracy is almost the same, but the epoch time dropped by half, my guess is that Visual Studio Code is the responsible for running the code slower. I will try the random initializer now.

ayrtondenner commented 6 years ago

After running the cmd command but updated with RandomNormal initializer, it got a great start: from 40% to 62% of accuracy in the first epoch, and already 84% accuracy in just the second epoch.

Step: 100
        Loss: 1.3824993634223939. Accuracy: 0.27787500485777855
  Step: 200
        Loss: 1.3787204509973525. Accuracy: 0.29250000447034835
  Step: 300
        Loss: 1.3749251115322112. Accuracy: 0.29779167100787163
  Step: 400
        Loss: 1.370024083852768. Accuracy: 0.306656254157424
  Step: 500
        Loss: 1.3648533897399902. Accuracy: 0.31392500472068785
  Step: 600
        Loss: 1.355485775868098. Accuracy: 0.32481250502169134
  Step: 700
        Loss: 1.3429256311484745. Accuracy: 0.33610714820878845
  Step: 800
        Loss: 1.3216396480798722. Accuracy: 0.35229688046500085
  Step: 900
        Loss: 1.298569918539789. Accuracy: 0.3684583390918043
  Step: 1000
        Loss: 1.2735181546211243. Accuracy: 0.38508750598132613
  Step: 1100
        Loss: 1.2506918907165527. Accuracy: 0.3988863695887002
  Step: 1200
        Loss: 1.2260725962618986. Accuracy: 0.41360417287796736
  Step: 1300
        Loss: 1.1971895770384715. Accuracy: 0.42967308334433113
  Step: 1400
        Loss: 1.173606401852199. Accuracy: 0.4422142926390682
  Step: 1500
        Loss: 1.1490636307001114. Accuracy: 0.4552750072578589
Epoch 0. Loss: 0.8477532631472537. Accuracy: 0.620921064050574
Epoch time: 0:03:31.892008. Total time: 0:03:33.047080

Epoch: 1
  Step: 100
        Loss: 0.7944310832023621. Accuracy: 0.6423750123381615
  Step: 200
        Loss: 0.7775250360369682. Accuracy: 0.651500013768673
  Step: 300
        Loss: 0.7654725861549377. Accuracy: 0.6531666805346806
  Step: 400
        Loss: 0.7508674088865519. Accuracy: 0.657781263589859
  Step: 500
        Loss: 0.7394263868927956. Accuracy: 0.6650500144958497
  Step: 600
        Loss: 0.727681430230538. Accuracy: 0.6728541818261147
  Step: 700
        Loss: 0.7172151552779334. Accuracy: 0.6816250161613737
  Step: 800
        Loss: 0.7068975845351815. Accuracy: 0.690093766823411
  Step: 900
        Loss: 0.6946409731772211. Accuracy: 0.6989444619417191
  Step: 1000
        Loss: 0.6850694700181484. Accuracy: 0.705600018799305
  Step: 1100
        Loss: 0.6746109608357603. Accuracy: 0.7131022923101078
  Step: 1200
        Loss: 0.6653454590340455. Accuracy: 0.7198229369024436
  Step: 1300
        Loss: 0.6552124836582404. Accuracy: 0.7267404058346382
  Step: 1400
        Loss: 0.6449800094110625. Accuracy: 0.7331250219685691
  Step: 1500
        Loss: 0.6353708243171374. Accuracy: 0.7386250225702922
Epoch 1. Loss: 0.4407074771429363. Accuracy: 0.8430263462819552
Epoch time: 0:03:30.585001. Total time: 0:07:04.885416

But in the end, it got the same rate of success, 86%~88% of test accuracy.

Epoch: 8
  Step: 100
        Loss: 0.15641237430274488. Accuracy: 0.9433750009536743
  Step: 200
        Loss: 0.15461750127375126. Accuracy: 0.9458750012516975
  Step: 300
        Loss: 0.1596740445246299. Accuracy: 0.9450833342472712
  Step: 400
        Loss: 0.1612383559346199. Accuracy: 0.9448750010132789
  Step: 500
        Loss: 0.16188879172503948. Accuracy: 0.944650001168251
  Step: 600
        Loss: 0.16811180506522458. Accuracy: 0.9424791690707207
  Step: 700
        Loss: 0.16696714759937353. Accuracy: 0.9430535733699799
  Step: 800
        Loss: 0.16778677925933153. Accuracy: 0.9426562521606684
  Step: 900
        Loss: 0.171104838690824. Accuracy: 0.9414027800824907
  Step: 1000
        Loss: 0.17126611874997616. Accuracy: 0.9410625022053719
  Step: 1100
        Loss: 0.17300944169136612. Accuracy: 0.9402386389537291
  Step: 1200
        Loss: 0.17448542379774154. Accuracy: 0.9396145863831044
  Step: 1300
        Loss: 0.17327917290994754. Accuracy: 0.9398076951962251
  Step: 1400
        Loss: 0.17414429432579448. Accuracy: 0.9395178604551724
  Step: 1500
        Loss: 0.17557645851373674. Accuracy: 0.9390416703224183
Epoch 8. Loss: 0.3664699006237482. Accuracy: 0.8835526535385534
Epoch time: 0:03:29.006842. Total time: 0:31:38.884601

Epoch: 9
  Step: 100
        Loss: 0.13474457811564208. Accuracy: 0.9521250003576278
  Step: 200
        Loss: 0.14127296243794263. Accuracy: 0.9500000002980232
  Step: 300
        Loss: 0.13752325549100836. Accuracy: 0.9521666653951009
  Step: 400
        Loss: 0.1373927940847352. Accuracy: 0.9524374970793724
  Step: 500
        Loss: 0.13874602177366616. Accuracy: 0.9518749974966049
  Step: 600
        Loss: 0.1388164148138215. Accuracy: 0.9514166640241941
  Step: 700
        Loss: 0.14371843845982638. Accuracy: 0.9495357124294553
  Step: 800
        Loss: 0.1437937137275003. Accuracy: 0.9493437480181456
  Step: 900
        Loss: 0.14414193288940522. Accuracy: 0.9494305535157521
  Step: 1000
        Loss: 0.1449411154780537. Accuracy: 0.9494249979853631
  Step: 1100
        Loss: 0.14623152853582394. Accuracy: 0.9488863619349219
  Step: 1200
        Loss: 0.14717813019485523. Accuracy: 0.9485729151467482
  Step: 1300
        Loss: 0.1493348607263313. Accuracy: 0.9474519222974778
  Step: 1400
        Loss: 0.15010015715046654. Accuracy: 0.9471249995061329
  Step: 1500
        Loss: 0.15107642836496235. Accuracy: 0.9466416665712992
Epoch 9. Loss: 0.4157790427145205. Accuracy: 0.868289503925725
Epoch time: 0:03:27.335671. Total time: 0:35:07.344262

With random initializer, it got a better start in the training, but it seems that even with that the network can't go past the previous rate of success.