activation function composer

[silky]

or more generally, a function composer

1. what does the graph of relu look like?
2. what about the graph of relu . tanh ?

and so on, indefinitely and arbitrarily.

some features:

• what points should be push through? maybe could add certain kinds of initialisations and ranges
• add things like drop-out and whatnot.

[silky]

using hoe and plot

hoe 'map (tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map (tanh . tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map ((\r -> if r > 0 r else 0) . tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map ((\r -> if r > 0 then r else 0) . tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map (tanh . (\r -> if r > 0 then r else 0) . tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map (tanh . (\r -> if r > 0 then r else 0) . tanh . tanh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map (tanh . (\r -> if r > 0 then r else 0) . tanh . sinh . (\x -> (x - 500)/50)) [0..1000]' | plot
hoe 'map (tanh . (\r -> if r > 0 then r else 0) . tanh . (\s -> exp s / (exp s + 1)) . (\x -> (x - 500)/50)) [0..1000]' | plot