RWKV-WORLD-4

[CosmoLM]

Model description

BlinkDL/rwkv-4-world is a repo present on Huggingface i want the model's tokenizer and the model to be added to the Transformers Lib.

Open source status

• [X] The model implementation is available
• [X] The model weights are available

Provide useful links for the implementation

No response

[amyeroberts]

Hi @CosmoLM, thanks for opening this model request!

The RWKV-4 model already exists in transformers -- PR, docs. To enable loading the model through Rwkv.from_pretrained, the checkpoints would need to be converted and model configs push to the hub using the conversion script.

I'd suggest opening a discussion on the hub to see if the repo owners would be interested in doing this.

[CosmoLM]

The RWKV-pile models are available but not the RWKV-world models because its tokenizer is not in the json format it is in txt format.

On Wed, 2 Aug, 2023, 4:24 pm amyeroberts, @.***> wrote:

Hi @CosmoLM https://github.com/CosmoLM, thanks for opening this model request!

The RWKV-4 model already exists in transformers -- PR https://github.com/huggingface/transformers/pull/22797, docs https://huggingface.co/docs/transformers/v4.31.0/en/model_doc/rwkv#rwkv-attention-and-the-recurrent-formulas. To enable loading the model through Rwkv.from_pretrained, the checkpoints would need to be converted and model configs push to the hub using the conversion script. https://github.com/huggingface/transformers/blob/8021c684ec3023295513be36bdc30e27e6f28cfc/src/transformers/models/rwkv/convert_rwkv_checkpoint_to_hf.py#L4

I'd suggest opening a discussion on the hub to see if the repo owners would be interested in doing this.

— Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/25253#issuecomment-1661993346, or unsubscribe https://github.com/notifications/unsubscribe-auth/BA7FALGYW7ERQ3LODEA6NADXTIWVPANCNFSM6AAAAAA3A3B6CY . You are receiving this because you were mentioned.Message ID: @.***>

[pranavsharma]

Is there an ETA for this?