RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.
Apache License 2.0
12.32k
stars
838
forks
source link
RWKV-v4neo expected scalar type BFloat16 but found Float #99
system: windows11+wsl2
lib: torch 2.0.0+cu118 CUDA 118 deepspeed 0.9.1,
python train.py --load_model "" --wandb "" --proj_dir "out" --data_file "./data_text_document" --data_type "binidx" --vocab_size 0 --ctx_len 128 --epoch_steps 1000 --epoch_count 20 --epoch_begin 0 --epoch_save 10 --micro_bsz 16 --n_layer 12 --n_embd 768 --pre_ffn 0 --head_qk 0 --lr_init 6e-4 --lr_final 1e-5 --warmup_steps 0 --beta1 0.9 --beta2 0.99 --adam_eps 1e-8 --accelerator cpu --devices 1 --precision bf16 --strategy ddp_find_unused_parameters_false --grad_cp 0
RUN CPU Out Error
RuntimeError: index_select(): self indexing axis dim should be positive