Open 00ffcc opened 1 month ago
可以看一下这里https://gitee.com/guizhiyu/rwkv_ascend
For RWKV, the time state is very sensitive, so fp16 can cause numerical overflow. Your approach is correct, but it will make the inference results different from what is expected.
For time-related calculations, due to exponential decay, precision is lost as values approach 0, rather than overflowing. Therefore we need to use fp32 for Time-related calculations.
FP16 overflow is addressed by x/2 every 6 layers.
However, this may have huge differences from original, therefore we have no plan to support it now.
我还真的试了一下在香橙派ai pro 16G上推理,有以下问题: