yuunnn-w / RWKV_Pytorch

This is an inference framework for the RWKV large language model implemented purely in native PyTorch. The official native implementation is overly complex and lacks extensibility. Let's join the flexible PyTorch ecosystem and open-source it together!
GNU General Public License v3.0
97 stars 7 forks source link

修复训练代码存在的潜在问题。 #20

Closed uniartisan closed 2 months ago

uniartisan commented 2 months ago
  1. 对不同序列的梯度规范化,解决权重的不一致
  2. 修复数据的设备同步