search

mlpen / Nystromformer

Apache License 2.0
356 stars 41 forks source link

Pre-trained weights #4

Closed NielsRogge closed 2 years ago

NielsRogge commented 3 years ago

Hi,

Thank you for this very clean repository! I wonder whether any checkpoints will be released? If yes, I'm interested in adding those to the HuggingFace Transformers library (the Reformer and LongFormer are already there, but the Linformer and the Nystromformer aren't yet). Since we only have to replace the self-attention mechanism, this seems quite straightforward in terms of modeling.

Kind regards,

Niels

yyxiongzju commented 3 years ago

Hi @NielsRogge, Thanks for your interest in Nystromformer. We are pretty packed recently for ICCV submission. We will release the checkpoint after the submission deadline. It will be great if you can help add to the HuggingFace Transformers.

yyxiongzju commented 3 years ago

Hi @NielsRogge , Thanks for your interest. I just finished some submissions. Here is the pretrained Nystromformer model on BookCorpus plus English Wikipedia for sequence length 512. I wonder if you can help add to the HuggingFace Transformers?

NielsRogge commented 3 years ago

Sure I can help! Do you have an email address to set up a Slack channel?

yyxiongzju commented 3 years ago

My email address is, yxiong43 at wisc.edu.

RobertHua96 commented 2 years ago

Hi, wondering if this was progressing? Would love to experiment with the pretrained model from huggingface!

novice03 commented 2 years ago

Nystromformer is now in huggingface: https://huggingface.co/docs/transformers/master/model_doc/nystromformer