search

tatp22 / linformer-pytorch

My take on a practical implementation of Linformer for Pytorch.
https://arxiv.org/pdf/2006.04768.pdf
MIT License
403 stars 36 forks source link

Huggingface #3

Closed flozi00 closed 4 years ago

flozi00 commented 4 years ago

Could you integrate this model into the huggingface transformers repository ? It would be a great addition there

tatp22 commented 4 years ago

Hi @flozi00! I am not done with my implementation right now, and I don't want to migrate stuff out of this repository until I am done with what I want to finish. When I am finished, I will investigate this further.

flozi00 commented 4 years ago

Great, I think the biggest task would be the formatting of the code.

Do you have experience about the model performance on squad question answering dataset ?

tatp22 commented 4 years ago

No clue. I am not using this self attention for this reason, so I don't think I would know.

There may be some templates for how to add new codes to help with that, but as I said, I will not look into this until I have implemented some more things.