search

weka511 / nlp

My experiments with Natural Language Processing. I've created a few programs to try out concepts.
GNU General Public License v3.0
1 stars 0 forks source link

Overflow during training #23

Closed weka511 closed 1 year ago

weka511 commented 1 year ago

./word2vec2.py train --N 100000 --tau 90000 ... Iteration= 4426, eta=0.0476, Loss=15260980.49 C:\Users\Weka\nlp\skipgram.py:292: RuntimeWarning: divide by zero encountered in log return - (np.log(expit(self.model.get_product(i_w,i_c_pos))) Iteration= 4451, eta=0.0476, Loss=inf Traceback (most recent call last): File "C:\Users\Weka\nlp\word2vec2.py", line 197, in optimizer.optimize() File "C:\Users\Weka\nlp\skipgram.py", line 375, in optimize raise Exception('Total loss overflow') Exception: Total loss overflow

weka511 commented 1 year ago

I assume that this is an attempt to calculate log(0), which occurs only if expit() receives an argument -inf, so the inner product of w and c must be infinite. In short, we already have an infinite value among the weights.

weka511 commented 1 year ago

I have desk checked the code, and think that the calcs are correct. Since I haven't seen this problem for small eta I plan to close shortly.

weka511 commented 1 year ago

Not seen