Add custom loss functions and a R/W state matrix

[voidvoxel]

Preface

I'm still documenting all of the changes, so please don't bonk me with the dumb-dumb-document stick! I promise to be a good girl and finish documenting all of the changes I made very shortly. I'll also make sure to add unit tests for the loss functions and R/W state matrices; I just need a moment to rest up and eat a meal first :revolving_hearts:

Description

I added support for defining custom loss functions, as well as a R/W state matrix to serve as RAM to allow for training by ruleset in addition to, or even in replacement of, a pre-existing set of training data. In my own benchmarks, a custom loss function designed to train for XOR improved training on the XOR data set by approximately 100% for NeuralNetwork (CPU) and 400% for NeuralNetworkGPU (GPU). This is, of course, a really specific example. However, it's just that; an example. This paves the way for any custom loss function to be defined. On GPU, you obviously are limited to values you pass through the NeuralNetwork.ram property, which is the R/W state matrix I previously mentioned. However, on CPU, you're honestly not limited by anything, as the function is called from the CPU, not the GPU, meaning you could hypothetically even go so far as to involve API calls in the loss function calculation if you really wanted to so long as your internet bandwidth permits without becoming too major of a bottleneck in training times.

Motivation and Context

discussion issue

How Has This Been Tested?

I ran the test suite. There's currently two failed tests that are related to these changes; the other 3-or-so failed tests are the same RNN tests that failed on a fresh copy of the master branch.

Of the 2 failed tests, the first one is related to the AE class; these changes actually break the current AE class due to how it was implemented. This is a non-issue, as custom loss functions give us a far better way to implement autoencoders.

The second failed test, however, I'm unsure of how to deal with it as of the current moment (likely a side effect of writing code daily for over a month on end with little food, water, or sleep because poverty is yikers and kind of makes surviving not so easy lol). I'll probably figure it out after some well-needed rest and food (I'm making a healthy meal for dinner rn to replenish my body :revolving_hearts:)

Screenshots (if appropriate):

Hyperfine benchmark results

nn-cpu-loss - NeuralNetwork with a custom loss function nn-cpu - NeuralNetwork without a custom loss function nn-loss - NeuralNetworkGPU with a custom loss function nn - NeuralNetworkGPU without a custom loss function

Why is the CPU so much faster?

Because the model used in the benchmarks is super tiny (2 inputs, 3 neurons in a single hidden layer, 1 output). Check out this issue for more information if you're interested.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [x] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Breaking changes

• The only breaking change is that the current autoencoder implementation will break. This is actually a neutral/good thing, as this feature adds a much better way to implement autoencoders anyways, making this singular breaking change a moot point seeing as we're still currently in a beta period before the next major release, and the AE class was literally just added not even a week or two ago. I'll be working on porting the new autoencoder implementation I wrote back into brain.js from the li'l sandbox I made for devtesting brain.js/gpu.js/related and hopefully it should be finished before pull time.
• The autoencoder class (AE) has been split into two classes: Autoencoder and AutoencoderGPU. Autoencoder extends NeuralNetwork and runs exclusively on the CPU. AutoencoderGPU extends NeuralNetworkGPU and runs on the GPU if one is present, and otherwise falls back to the CPU.

Author's Checklist:

• [x] My code focuses on the main motivation and avoids scope creep.
• [ ] My code passes current tests and adds new tests where possible.
• [x] My code is SOLID and DRY.
• [ ] I have updated the documentation as needed.

Reviewer's Checklist:

• [ ] I kept my comments to the author positive, specific, and productive.
• [ ] I tested the code and didn't find any new problems.
• [ ] I think the motivation is good for the project.
• [ ] I think the code works to satisfies the motivation.

[voidvoxel]

Working on the failed CI lint tasks rn while my food is in the microwave!! I'm sorry, I'm just so excited for brain.js to have this feature!!! I've been wishing we could do this in brain for literal years now :smiling_face_with_three_hearts: :two_hearts:

[voidvoxel]

All is well except for that one CI / Build error for Mac :two_hearts: :two_hearts: :two_hearts: :blush: :two_hearts: :two_hearts: :two_hearts:

[voidvoxel]

Many apologies for taking a brief hiatus! A lot has happened in my personal life. I'm starting work on implementing these features within the RNN family of classes in the API tonight :two_hearts:

[robertleeplummerjr]

Could you clean up conflicts, and we can get this in asap? Thank you for your patience.

[voidvoxel]

Could you clean up conflicts, and we can get this in asap? Thank you for your patience.

Absolutely!! I'm so sorry that I've taken so long; my health hasn't been the best. I'm supposed to go in for surgery tomorrow, so I'll see if I can rush it tonight :blush: If not, I'll make sure to resolve them as soon as I'm in working condition again, which shouldn't take more than a few days :relaxed:

[voidvoxel]

My health is permitting near-full-time work again. I'll update you once we're ready for merging :blush: