mem and norm_term is nan？

DavideHe commented 4 months ago

I loaded the model and run the inference just , I found mem and norm_term are large and round 2nd is inf . """ [Update] self.norm_term 7444.0 1181.0 [Update] self.memory 11912.0 -11120.0 [Update] self.norm_term 6420.0 1126.0 [Update] self.memory 13560.0 -14040.0 [Update] self.norm_term 7524.0 1179.0 [Update] self.memory 13808.0 -12528.0 [Update] self.norm_term 6344.0 1184.0 [Update] self.memory 18416.0 -15440.0 [Update] self.norm_term 8456.0 613.5 [Update] self.memory 23856.0 -26608.0 [Update] self.norm_term 8648.0 964.5 [Update] self.memory 25968.0 -26096.0 [Update] self.norm_term 12440.0 175.875 Loss @ segment 0: 13.347726821899414 --------------------------------------------- Round 0 ---------------------------------------------- [Update] self.memory inf -inf [Update] self.norm_term 46080.0 586.0 [Update] self.memory nan nan [Update] self.norm_term nan nan """ values are max and min of the tensor. this is just value error.

ZackZikaiXiao commented 4 months ago

Loss @ segment 0: 2.907336950302124 Loss @ segment 0: 3.4686388969421387 Loss @ segment 0: 1.9305295944213867 Loss @ segment 0: 3.291438579559326 Loss @ segment 0: 1.6723694801330566 Loss @ segment 0: 3.4830915927886963 Loss @ segment 0: 3.3260433673858643 Loss @ segment 0: 3.103365898132324 0%| | 1/13201 [00:08<31:18:20, 8.54s/it]Step: 1, Loss: 3.291438579559326, LR: 9.999242431416371e-05 Step: 1, Loss: 3.4830915927886963, LR: 9.999242431416371e-05 Step: 1, Loss: 2.907336950302124, LR: 9.999242431416371e-05Step: 1, Loss: 3.4686388969421387, LR: 9.999242431416371e-05 Step: 1, Loss: 3.3260433673858643, LR: 9.999242431416371e-05 Step: 1, Loss: 1.6723694801330566, LR: 9.999242431416371e-05 Step: 1, Loss: 3.103365898132324, LR: 9.999242431416371e-05 Step: 1, Loss: 1.9305295944213867, LR: 9.999242431416371e-05 Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan Loss @ segment 0: nan

I experience the same issue; it does not consistently occur, but it seems that the likelihood of it happening increases with larger block size. How should this be resolved?

DavideHe commented 4 months ago

Did you load llama3-8b pretrain model?

LWL-cpu commented 3 months ago

May I ask if you have solved this problem? I have also encountered this problem

DavideHe commented 3 months ago

if you use transformers add gate parameters , you should init gate parameters on function _init_weights of LlamaPreTrainedModel; because transformers init parameters with device meta. this device just init weight with shape without real values. So the parameters will init likely empty values when it get the values on GPU . If you want to comfirm this above, you can print the gate when you start training on the GPU. You will find the gate are -1e-30 ~ 1e30 that is very large values.

LWL-cpu commented 2 months ago

I apologize for the delay in seeing your reply, but I couldn't find this function: _init_weights.

DavideHe commented 2 months ago

https://github.com/Beomi/InfiniTransformer/blob/main/infini_llama/modeling_infini_llama.py#L1297

redefine PreTrainedModel class function _init_weights

WF0511 commented 2 months ago

这部分具体是如何设置的呢，是否可以share一下~感激不尽

LWL-cpu commented 2 months ago

您好，我现在还是有这个问题。在训练gemma的时候，发现是memory本身计算出现inf，并不是gate导致的。请问怎么解决呢

WF0511 commented 2 months ago

这是来自QQ邮箱的假期自动回复邮件。邮件已收到。

DavideHe commented 2 months ago

Modify code in here. Add some condition about gate params.

DavideHe commented 2 months ago

这个就需要具体问题具体分析了，最简单的办法就是打印日志，看在哪个位置最先开始出现nan，不能只看最后的输出。我看了下这里面能引入nan的，最大可能性还是gate引入。其他的地方概率不大，因为norm_term已经缩小了mem的幅值带来的不断增长的问题。

LWL-cpu commented 2 months ago

感谢您的回复。我打印了一下是KV QM这些相乘导致矩阵的某些值过大，出现了inf，然后导致NAN。我也不知道怎么回事。

LWL-cpu commented 2 months ago

我看代码里mem和norm_term是分开算并且累加的哇，所以每项不会特别大吗

DavideHe commented 2 months ago

是会变大，你可以把这块代码优化一下，但是以32k seq len ，2k的segsize，bf16来训练，至少不会nan

LWL-cpu commented 2 months ago

我用的这个代码，在minipile数据集上训练的gemma-2b,然后打印的gate确实都是-100，第三四个segment的loss就是nan了，然后我打印发现有些层的memory变成inf了。我目前用的lora微调，把memory部分去掉是可以正常的。

Beomi / InfiniTransformer

mem and norm_term is nan？ #21