search

syncdoth / RetNet

Huggingface compatible implementation of RetNet (Retentive Networks, https://arxiv.org/pdf/2307.08621.pdf) including parallel, recurrent, and chunkwise forward.
MIT License
226 stars 24 forks source link

HuggingFace checkpoint #36

Open xtwigs opened 10 months ago

xtwigs commented 10 months ago

Hi! Congrats on the clean RetNet code. I'm experimenting with the torchscale codebase and happened to find your repo with a link to a checkpoint in HF.

I noticed it is now gone, do you have plans to reupload it? or further details on the training dataset/hyperparameters?

Thanks in advance :)

syncdoth commented 10 months ago

Hi there! Sorry for new year delay. Last time we uploaded the model, it was quite under-trained and the performance was not as good as expected. We are still training the model to make sure it's safe to deploy. Will also ping here when the model goes live again.

xtwigs commented 10 months ago

Thanks for the reply! I'm working with the RetNet architecture in a downstream task, so a base pretrained model would definitely help out.