novak-99
commented
3 years ago Thank you so much for finding this issue, I never thought about optimization in that manner!
I just pushed the commit and changed the softmax function with that optimized implementation.
https://github.com/novak-99/MLPP/blob/4ebcc0a1e3866e54d61933dcdbe4d5ce902ff6a7/MLPP/Activation/Activation.cpp#L52-L64
Here is one specific example of code optimization: The softmax function here will give the correct answer but as it is currently written it is recalculating the same sum z.size() times. It would be much better to calculate the sum outside of the loop once and then reuse that value inside the loop without recalculating it.
If we wanted to optimize even more we could look at the use of the exp() function. Even with the above fix the exponential of each element in z is being calculated twice (once for the sum and once for the final output element calculation). Assuming memory allocations and accesses are faster than the exp() function it would be better to make an intermediary array of the exponential values and then access that array to calculate the sum and then also to calculate values of a.
The first optimization here will make a massive difference, so I think it is definitely worth keeping things like this in mind. The second one will have much less of an impact and goes a bit more into the weeds, so I would not worry too much about optimizations like that during the initial coding - I mention it here just to give a more complete idea of what kind of optimizations are possible even in a very simple function.