Fix softmax overflow - Githubissues

numenta / nupic-legacy

Numenta Platform for Intelligent Computing is an implementation of Hierarchical Temporal Memory (HTM), a theory of intelligence based strictly on the neuroscience of the neocortex.

http://numenta.org/

GNU Affero General Public License v3.0

6.33k stars 1.56k forks source link

Fix softmax overflow #3819

Closed suzusuzu closed 6 years ago

suzusuzu commented 6 years ago

Please look this softmax formula.

$\frac{e^{x_i}}{\sum_j{e^{x_j}}} = \frac{e^{-m}}{e^{-m}} \cdot \frac{e^{x_i}}{\sum_j{e^{x_j}}} = \frac{e^{x_i-m}}{\sum_j{e^{x_j-m}}}$

If x_i is too big, it occur overflow. It is necessary to subtract constant m. Sometimes people set m to be the maximum of x.

rhyolight commented 6 years ago

@scottpurdy Can you take a look at this?

scottpurdy commented 6 years ago

Edited: @lscheinkman looked this up and apparently this is a common way to fix it. This change looks good but can you add a unit test for this function?