numerical instability in free_energy

[limbo-wg]

Hello~

Thank you very much for sharing the code, which really helps me a lot.

But I find some problem when I use the code in

RBM.ipynb

def free_energy(self,v):
    vbias_term = v.mv(self.v_bias)
    wx_b = F.linear(v,self.W,self.h_bias)
    hidden_term = wx_b.exp().add(1).log().sum(1)
    return (-hidden_term - vbias_term).mean()

I found that there may be numerical instability in free_energy. In hidden_term = wx_b.exp().add(1).log().sum(1), if wx_b contains elements that greater than or equal to 88, the result of wx_b.exp() is inf. And this may cause the result of loss to become NAN.

It could be fixed by making the following change:

hidden_term = wx_b.exp().clamp(max=87).add(1).log().sum(1)

Hope to hear from you ~

Thanks in advance! : )

[Justobe]

ping~ @odie2630463

[odie2630463]

Hi , I am so sorry for replying late, I think it's numerical issue cause by log. Using clamp maybe a way to solve , but I will check if there have other implements and update it. (Old project to me haha)

Thanks you point out this issue!