colgreen
commented
6 years ago You may be interested in the vectorized neural net implementations here:
NeuralNet/Double/Vectorized NeuralNet/Double/ActivationFunctions/Vectorized
That code is using the SIMD/vector CPU instructions in most modern x86 CPUs. The next big jump in speed will come from moving from double to single precision floats (from 64bit to 32bit).
A significant issue with CUDA is that it is necessary to transfer the neural net data from main memory to GPU memory, and that will likely mitigate any performance improvements. It may be possible to overcome that problem by keeping genome and neural net data on the GPU, but that would be a whole new project I think (CUDANeat).
Also note that the CPPNs in HyperNEAT allow for each neuron to specify which activation function is uses (from an available set), thus limiting attempt to vectorize the activation function calculations in CPPNs.
If you don't need/want HyperNEAT then it shouldbe possible to add the above linked neural net code to SharpNEAT 2.x, add a genome decoder to build the neural net instances from a sharpneat 2.x genome, and that should do it. One thing to note is that there is no concept of a bias node in the sharpneat refactor - instead the tasks should define one extra input and provide a fixed 1.0 input, which is all a bias node is, it's just that it's not handled as a special case anymore.
How can I re-write Activate method for CUDA? The CPU computations are pretty slow