search

cazala / synaptic

architecture-free neural network library for node.js and the browser
http://caza.la/synaptic
Other
6.92k stars 666 forks source link

Stops after one iteration. #179

Closed snowfrogdev closed 7 years ago

snowfrogdev commented 7 years ago

I'm running a 15-8-1 perceptron and it always stops after one iteration. My training set has 38 entries. Here is a sample from my training set:

{
    input: [-1, 1, -1, -1, -1, 1, -1, 1.101903144, -0.524624152, 0.484064429, 1.251901816, -0.925710813, -1, -1, -1], 
    output: [281500]
}

Here is my training function call:

trainer.train(trainingSet,{
    rate: 0.1,
    iterations: 20,
    error: 0.1,
    shuffle: true,
    log: 1,
    cost: Trainer.cost.CROSS_ENTROPY
});

And here is the log I get:

"iterations" 1 "error" -7206655.541984386 "rate" 0.1

I have tried adding hidden layers, adding or removing neurons from the hidden layer(s), raising and lowering the error parameter as well as the rate and I always get approximately the same result with only one iteration. I even tried normalizing the output in the training set and I would still get similar (normalized) and it still stopped after only one iteration.

Here is the link to the codepen http://codepen.io/Neoflash1979/pen/rjNYLN/?editors=1011

If anyone could help me out with this problem it would be much appreciated.

DanielJohnBenton commented 7 years ago

I'm not seeing this issue in the code you linked?

"iterations" 5000 "error" 0.538447341879756 "rate" 0.05
"iterations" 10000 "error" 0.5378594861187836 "rate" 0.05
"iterations" 15000 "error" 0.5373291753169468 "rate" 0.1
"iterations" 20000 "error" 0.5368596959527838 "rate" 0.1
"iterations" 25000 "error" 0.5367229011520571 "rate" 0.15
"iterations" 30000 "error" 0.5366305968300676 "rate" 0.15
"iterations" 35000 "error" 0.5366687526843594 "rate" 0.15
"iterations" 40000 "error" 0.5366272299146678 "rate" 0.2
"iterations" 45000 "error" 0.5366396865669854 "rate" 0.2
"iterations" 50000 "error" 0.5366466071966102 "rate" 0.2
196711.5328826183
snowfrogdev commented 7 years ago

@DanielJohnBenton I have, after several hours :-) of tinkering, figured out, I think, why it would stop after one iteration. It seems to be related to the fact that my inputs were z-score normalized around 0. After fixing that and scaling my inputs in the 0-1 range, it would still stop after one iteration. I then changed my outputs from raw data(numbers in the 100 000s) and also scaled it in the 0-1 range and that did the trick. Although I can't seem to configure the network to find a solution with a suitable error rate, yet.

I don't understand why but it seems like the Network can't take negative values or is it that it can't any values outside the 0 -1 range? Much of what I'm reading about Multi-layer Perceptron suggests that it is preferable to normalize your data in the -1 to 1 range and that it's often even better to z-score normalize rather than scale normalize. Does Synaptic not allow for that?

DanielJohnBenton commented 7 years ago

I'm getting somewhere close with these settings:

trainer.train(trainingSet,{
    rate: [0.06, 0.03, 0.01, 0.005],
    iterations: 600000,
    error: 0.00000001,
    shuffle: true,
    log: 5000,
    cost: Trainer.cost.MSE
})
//.then(results => console.log('done!', results));

console.log(myNetwork.activate([1, 0, 0, 0, 1, 0, 0, 0.631578947, 0.375, 0, 0.25, 0.333333333, 1, 1, 1])); // should be 0.46
console.log(myNetwork.activate([0, 1, 0, 0, 1, 0, 0, 0.421052632, 0.25, 0.2, 0, 0.333333333, 0, 0, 0])); // should be 0.21
console.log(myNetwork.activate([1, 0, 0, 0, 1, 0, 0, 0.473684211, 0.375, 0, 0.25, 0.333333333, 1, 0, 1])); // should be 0.37
iterations 5000 error 0.0009128241164576313 rate 0.06
iterations 10000 error 0.0006627641098215922 rate 0.06
iterations 15000 error 0.00048793509655490464 rate 0.06
iterations 20000 error 0.00043792661622451374 rate 0.06
iterations 25000 error 0.00041132177755413504 rate 0.06
iterations 30000 error 0.00039142879229006773 rate 0.06
iterations 35000 error 0.0003735074892307478 rate 0.06
iterations 40000 error 0.0003663756541162279 rate 0.06
iterations 45000 error 0.0003573366757346837 rate 0.06
iterations 50000 error 0.00035204309283617415 rate 0.06
iterations 55000 error 0.0003435048635319246 rate 0.06
iterations 60000 error 0.00033440708356446004 rate 0.06
iterations 65000 error 0.0003265361093365113 rate 0.06
iterations 70000 error 0.0003206072208469512 rate 0.06
iterations 75000 error 0.0003155589746379591 rate 0.06
iterations 80000 error 0.0003121626983804175 rate 0.06
iterations 85000 error 0.0003056378711758537 rate 0.06
iterations 90000 error 0.00030145537163213994 rate 0.06
iterations 95000 error 0.0002994057530289927 rate 0.06
iterations 100000 error 0.00029399542119881017 rate 0.06
iterations 105000 error 0.00028406581507690876 rate 0.06
iterations 110000 error 0.00028490235063567183 rate 0.06
iterations 115000 error 0.00027323938776665493 rate 0.06
iterations 120000 error 0.00027883340112782744 rate 0.06
iterations 125000 error 0.0002735952370865265 rate 0.06
iterations 130000 error 0.00026745829280689287 rate 0.06
iterations 135000 error 0.00025739942693719865 rate 0.06
iterations 140000 error 0.0002612619537172321 rate 0.06
iterations 145000 error 0.00024205378389293992 rate 0.06
iterations 150000 error 0.00024025361454100644 rate 0.03
iterations 155000 error 0.00023473639470630473 rate 0.03
iterations 160000 error 0.00023290368300965216 rate 0.03
iterations 165000 error 0.00023418303418775138 rate 0.03
iterations 170000 error 0.0002289552596481291 rate 0.03
iterations 175000 error 0.00022937494870163948 rate 0.03
iterations 180000 error 0.00021954182295593817 rate 0.03
iterations 185000 error 0.00022216636632428017 rate 0.03
iterations 190000 error 0.00022475079865222824 rate 0.03
iterations 195000 error 0.00022290291015986623 rate 0.03
iterations 200000 error 0.00022024042877391605 rate 0.03
iterations 205000 error 0.00021727096071648126 rate 0.03
iterations 210000 error 0.0002149624469807011 rate 0.03
iterations 215000 error 0.00021446953848904904 rate 0.03
iterations 220000 error 0.00021260781916275856 rate 0.03
iterations 225000 error 0.00021141949287962106 rate 0.03
iterations 230000 error 0.0002090720009544603 rate 0.03
iterations 235000 error 0.00020972880193943742 rate 0.03
iterations 240000 error 0.00020651700025314446 rate 0.03
iterations 245000 error 0.00019981142514097677 rate 0.03
iterations 250000 error 0.0002056051477826538 rate 0.03
iterations 255000 error 0.00020317837485421444 rate 0.03
iterations 260000 error 0.0002030443884111394 rate 0.03
iterations 265000 error 0.00020263276712341433 rate 0.03
iterations 270000 error 0.00020128876430334418 rate 0.03
iterations 275000 error 0.00019357599803853723 rate 0.03
iterations 280000 error 0.00019969200541805997 rate 0.03
iterations 285000 error 0.00019778496603033453 rate 0.03
iterations 290000 error 0.00019828751877976817 rate 0.03
iterations 295000 error 0.00018992671542394015 rate 0.03
iterations 300000 error 0.00019178568824840196 rate 0.01
iterations 305000 error 0.0001903005476507752 rate 0.01
iterations 310000 error 0.0001897532373532436 rate 0.01
iterations 315000 error 0.00018992394001692052 rate 0.01
iterations 320000 error 0.0001900640564259115 rate 0.01
iterations 325000 error 0.00018997706841250388 rate 0.01
iterations 330000 error 0.0001889014162789612 rate 0.01
iterations 335000 error 0.0001891870571297435 rate 0.01
iterations 340000 error 0.0001893173119363351 rate 0.01
iterations 345000 error 0.0001883940343837463 rate 0.01
iterations 350000 error 0.00018804842706317528 rate 0.01
iterations 355000 error 0.0001881750898845622 rate 0.01
iterations 360000 error 0.00018777654766604488 rate 0.01
iterations 365000 error 0.0001880043470242239 rate 0.01
iterations 370000 error 0.00018744412577501358 rate 0.01
iterations 375000 error 0.00018728106247633755 rate 0.01
iterations 380000 error 0.00018620565835659057 rate 0.01
iterations 385000 error 0.0001865327544463781 rate 0.01
iterations 390000 error 0.0001854397070096246 rate 0.01
iterations 395000 error 0.00018644226929323546 rate 0.01
iterations 400000 error 0.00018537120867098686 rate 0.01
iterations 405000 error 0.00018550291085422883 rate 0.01
iterations 410000 error 0.0001842731263756881 rate 0.01
iterations 415000 error 0.00018431947463075777 rate 0.01
iterations 420000 error 0.0001846126474855285 rate 0.01
iterations 425000 error 0.0001839495153502337 rate 0.01
iterations 430000 error 0.0001843834469396247 rate 0.01
iterations 435000 error 0.00018363443458750716 rate 0.01
iterations 440000 error 0.00018340273388240088 rate 0.01
iterations 445000 error 0.00018222465768087258 rate 0.01
iterations 450000 error 0.00018172159262521786 rate 0.005
iterations 455000 error 0.0001813161692966999 rate 0.005
iterations 460000 error 0.00018060255847728184 rate 0.005
iterations 465000 error 0.00018084521077790588 rate 0.005
iterations 470000 error 0.00018084482934121988 rate 0.005
iterations 475000 error 0.00018070418364636364 rate 0.005
iterations 480000 error 0.00018012800824365155 rate 0.005
iterations 485000 error 0.00018028592531442596 rate 0.005
iterations 490000 error 0.00018027706640859085 rate 0.005
iterations 495000 error 0.00018020082991337263 rate 0.005
iterations 500000 error 0.00017977654104506688 rate 0.005
iterations 505000 error 0.00017976564018955203 rate 0.005
iterations 510000 error 0.00017918721511560364 rate 0.005
iterations 515000 error 0.0001793573934100284 rate 0.005
iterations 520000 error 0.00017926913466883107 rate 0.005
iterations 525000 error 0.0001786674951670656 rate 0.005
iterations 530000 error 0.00017904332981124866 rate 0.005
iterations 535000 error 0.00017888413768209685 rate 0.005
iterations 540000 error 0.00017870093446470392 rate 0.005
iterations 545000 error 0.00017853022182001918 rate 0.005
iterations 550000 error 0.00017830020731072444 rate 0.005
iterations 555000 error 0.00017816752582574188 rate 0.005
iterations 560000 error 0.0001778976517462462 rate 0.005
iterations 565000 error 0.00017791121308663772 rate 0.005
iterations 570000 error 0.0001776620116186246 rate 0.005
iterations 575000 error 0.00017737744430677444 rate 0.005
iterations 580000 error 0.00017734805075877427 rate 0.005
iterations 585000 error 0.00017703539744787954 rate 0.005
iterations 590000 error 0.00017693685924924845 rate 0.005
iterations 595000 error 0.00017675104743122169 rate 0.005
iterations 600000 error 0.00017674797061641335 rate 0.005
[ 0.45968149663616875 ]
[ 0.2220596482013724 ]
[ 0.3662527911734674 ]
128302.04281890606
snowfrogdev commented 7 years ago

Wow, you're results are so much better than what I was getting. I guess it's the switch to the MSE cost function that did the trick. I had tried MSE cost before but the training wouldn't run at all when I did that. But when I also change the learning rate and minimum error like you did, I get pretty good results. I did you know that lowering the learning rate by a factor of ten would work?

DanielJohnBenton commented 7 years ago

Just a bit of intuition and some trial and error. I'm still learning about this stuff. :^) The number of iterations was also important which kind of limits you in some ways as it affects how many runs at X learning rate occur (if this is too limiting, you could perform the training in multiple batches or something).

I noticed a couple problems in your earliest copypastes that wouldn't have helped: