CodingTrain / Toy-Neural-Network-JS

Neural Network JavaScript library for Coding Train tutorials
MIT License
425 stars 242 forks source link

Add more Activation Function options. #75

Open Adil-Iqbal opened 6 years ago

Adil-Iqbal commented 6 years ago

As per this discussion: #70

This pull request adds the following features to the library.

My javascript auto-formatter cleaned up the code, but it obscured the relavent changes in the most recent commit. It might be more informative to look at an older commit...

xxMrPHDxx commented 6 years ago

Lol. Just exploring your code to add more activation function into my code and found that out. Now all of them should work fine

Adil-Iqbal commented 6 years ago

Ran some more tests on XOR. Everything works great! There was a 100% pass rate and most of them were quick. Observations below:

  1. leaky_relu had the best balance of consistency and speed.
  2. relu was the fastest, but not consistent. Still exhibiting "dying relu" problem.
  3. relu was generating an interesting step patterns in the solution space. Here's a picture.
  4. arctan was solving the problem consistently, albeit not as fast as the others. The solution space looked like a large open field with folded corners. Here's a picture.
  5. softplus had the most difficult time. But it consistently solved the problem. Lots of fluctuations and BOLD decision making. The solution space looks like a glowing blob! Here's a picture.

I encourage everyone to check it out. It's really interesting!