How to write the RWKV in autogressive style like RNN

BlinkDL / ChatRWKV

ChatRWKV is like ChatGPT but powered by RWKV (100% RNN) language model, and open source.

Apache License 2.0

9.39k stars 689 forks source link

Open HaiFengZeng opened 11 months ago

HaiFengZeng commented 11 months ago

instead of running in a teacher-forcing in training stage, can we run the model in free running style?:

for i in range(T):
    X_{t}=RNN(X_{t-1},hidden_state)

since teacher-forcing may have an exposure bias problem.

BlinkDL commented 11 months ago

yes you can

BlinkDL commented 11 months ago