search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
131.71k stars 26.22k forks source link

FastFormers into transformers #8083

Closed ykim362 closed 2 years ago

ykim362 commented 3 years ago

🌟 New model addition

Model description

We just open-sourced FastFormers which are our SustaiNLP 2020 systems (FastFormers: Highly Efficient Transformer Models for Natural Language Understanding paper).

Currently, we are hosting this on our repository, but would like to merge it back to the transformers repository as an example. our repo - https://github.com/microsoft/fastformers

For the purpose of the shared task, this is purely implemented with SuperGLUE data set. So, it's dependent on Alex Wang(@W4ngatang)'s SuperGLUE data processing pipeline. Also, many parts of the implementations are based on Alex'. (https://github.com/W4ngatang/transformers/tree/superglue)

What would be the best way to merge this back?

Open source status

LysandreJik commented 3 years ago

Hi! This is great, thanks for offering to contribute it! From what I understand, FastFormers contains several scripts that can be applied to transformers models out of the box, that is, training, distillation, pruning, using quantization alongside the onnx runtime and fp16 optimizations.

Is that correct? If that is so, the easiest way would be to add the corresponding scripts to the examples/ directory, probably under examples/fastformers. If there are modifications made to the model themselves, we can take a look together at how we can integrate those in the library.

JetRunner commented 3 years ago

Hi, thanks for your interest! From what I understand, I think your model falls in the category of dynamic acceleration. For these types of paper, I recommend you to integrate it to examples/, just like PABEE and DeeBERT. I've emailed you an invitation to our Slack channel if it works for you. cc @LysandreJik

ykim362 commented 3 years ago

@LysandreJik yes, that is correct. Thanks @JetRunner, let's discuss more on the slack.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

ykim362 commented 2 years ago

I am sorry, but I have been fully loaded with some other stuffs. I won't be able to make a progress. I'd like to close this to avoid any confusion.