Encountered runtime warning while computing logistic function

[vimalthilak]

@karpathy Thanks for open sourcing your image-to-sentences work. I got the code up & running with the Flickr30K dataset but encountered a runtime warning " RuntimeWarning: overflow encountered in exp"

I have fixed it locally by using scipy.special.expit function. I have attached the patch below in case you want to "cherry-pick' my commit. Let me know if this patch is useful to you and whether you'd like me to make a PR with a fix:

From d3b8d3401a7ebeae1aff88538f1f5eff440b31cf Mon Sep 17 00:00:00 2001 From: Vimal Thilak Date: Wed, 3 Dec 2014 15:16:28 -0800 Subject: [PATCH] [bugfix] Fix overflow runtime warning

• Warning encountered in logistic function computation

  Signed-off-by: Vimal Thilak

  imagernn/lstm_generator.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/imagernn/lstm_generator.py b/imagernn/lstm_generator.py index 011e333..af6797f 100644 --- a/imagernn/lstm_generator.py +++ b/imagernn/lstm_generator.py @@ -1,5 +1,6 @@ import numpy as np import code +import scipy.special

from imagernn.utils import initw

@@ -75,7 +76,7 @@ class LSTMGenerator: IFOG[t] = Hin[t].dot(WLSTM)

   # non-linearities

• IFOGf[t,:3_d] = 1.0/(1.0+np.exp(-IFOG[t,:3_d])) # sigmoids; these are the gates

• IFOGf[t,:3*d] = scipy.special.expit(IFOG[t, :3*d])  #1.0/(1.0+np.exp(-IFOG[t,:3*d])) # sigmoids; these are the gates

  IFOGf[t,3_d:] = np.tanh(IFOG[t, 3_d:]) # tanh

  compute the cell activation

  @@ -224,7 +225,7 @@ class LSTMGenerator: C = np.zeros((1, d)) Hout = np.zeros((1, d)) IFOG[t] = Hin[t].dot(WLSTM)

• IFOGf[t,:3_d] = 1.0/(1.0+np.exp(-IFOG[t,:3_d]))
• IFOGf[t,:3_d] = scipy.special.expit(-IFOG[t,:3_d]) # 1.0/(1.0+np.exp(-IFOG[t,:3_d])) IFOGf[t,3_d:] = np.tanh(IFOG[t, 3_d:]) C[t] = IFOGf[t,:d] * IFOGf[t, 3_d:] + IFOGf[t,d:2d] \ c_prev if tanhC_version: -- 2.0.1

[karpathy]

Hi thank you for reporting this. The exp should be guarded, you're right, but despite the warning I don't believe this should be too big of an issue. I will fix it soon, I'm overwhelmed with other work right now. Thank you!

[amangupta199334]

@vimalthilak , Thank you sir. It really removed the warning. But, as @karpathy sir said, warning does't make any trouble. Anyways, good job sir.

[monajalal]

@karpathy I changed my sigmoid to expit from scipy.special to overcome the overflow but now I am getting overflow somewhere else.

/Users/mona/machine_learning/pset4/mlp.py:304: RuntimeWarning: overflow encountered in exp
  return expit(-x)

What should I use?

Would it make sense to replace sigmoid with tanh?

def logistic_function(x):
    return .5 * (1 + np.tanh(.5 * x))