Create Neuron.Activation class or factory

[levithomason]

Currently there is an activation namespace. Its members are activation function definitions. These are objects with func, prime, rangeMin, and rangeMax properties.

These objects should be created by a class or factory so they can be validated, extended, and created consistently. For better cohesion, this should be a static Neuron class or factory. Same for the namespace, it should likely be a Neuron namespace.

We should likely end up with something along the lines of:

// current namespace with its activations
Neuron.ACTIVATION.tanh

// create new activation
Neuron.ACTIVATION.create({
  name: 'newActivation',
  func: x => x,
  prime: x => x,
  rangeMin: 0,
  rangeMax: 1,
})

// => Neuron.ACTIVATION.newActivation

EDIT Also, finish the docs in the Neuron class while at this. There are missing and incomplete doc strings here.

[ckcollab]

From my understanding that sounds good.

Will there be an easy way to iterate over them? Neuron.ACTIVATION.get_all() or what have you? May not be helpful or needed.

Also, is there a way to "choose" one for you? Neuron.ACTIVATION.find_a_good_fit(some_parameters_i_dont_understand)

[levithomason]

Since it is just an object you can loop over them like:

// lodash
_.each(Neuron.Activation, (activation, name) => {...})

// vanilla js
Object.keys(Neuron.Activation).forEach(key => {
  Neuron.Activation[key]
})

// or
for (const key in Neuron.Activation) {
  if (Neuron.Activation.hasOwnProperty(key) {
    // do stuff
  }
}

There are possibly heuristics we can use to try and find an activation function for you. It might require asking some questions or something. Realistically, I think knowing what they do and what they are good at is essential to good machine learning. Also, experimenting is really fun and another great way to findTheBestFunction :)

[ckcollab]

:+1: thanks for explaining

So you'd make _.each ignore create()? Or does _.each only cover objects and not functions... but I thought functions are objects... how does that work? :)

[levithomason]

You're right, in these cases it would totally pick up create(). That is what I get for coding at this hour :P The implementation of this fix will be something different, but the goal is the same and remains good. Split out the activation function definition somewhere, validate it, and allow adding new ones.

Thanks for the dive in and catching that. Night night :full_moon_with_face: