search

lucidrains / reformer-pytorch

Reformer, the efficient Transformer, in Pytorch
MIT License
2.1k stars 254 forks source link

Pretrained models? #71

Closed Hellisotherpeople closed 4 years ago

Hellisotherpeople commented 4 years ago

When are we going to see pretrained reformer models? I don't have the compute or the dataset for doing it myself - but this seems to be strictly better of a technique for training NLP models than previous transformers

lucidrains commented 4 years ago

There isn't any. You should go make one! Perhaps use the new fancy Electra? https://openreview.net/forum?id=r1xMH1BtvB