Fix softmax overflow - Githubissues

numenta / nupic-legacy

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

http://numenta.org/

GNU Affero General Public License v3.0

6.34k stars 1.56k forks source link

Fix softmax overflow #3818

Closed suzusuzu closed 6 years ago

suzusuzu commented 6 years ago

Please look this softmax formula.

$\frac{e^{x_i}}{\sum_j{e^{x_j}}} = \frac{e^{-m}}{e^{-m}} \cdot \frac{e^{x_i}}{\sum_j{e^{x_j}}} = \frac{e^{x_i-m}}{\sum_j{e^{x_j-m}}}$

If x_i is too big, it occur overflow. It is necessary to subtract constant m. Sometimes people set m to be the maximum of x.

suzusuzu commented 6 years ago

Sorry, I didn't know the Contributor License. Now, I have signed the Contributor License.