Can Torch-Pruning be applied to Transformer models ?

VainF / Torch-Pruning

[CVPR 2023] Towards Any Structural Pruning; LLMs / SAM / Diffusion / Transformers / YOLOv8 / CNNs

https://arxiv.org/abs/2301.12900

MIT License

2.59k stars 318 forks source link

Open SefaZeng opened 3 years ago

SefaZeng commented 3 years ago

I try to follow the example to prune the transformer model, but the layer-norm always gets a size mismatch error.

VainF commented 3 years ago

Hi @SefaZeng , layer-norm is unfortunately not supported yet. We are working on that to support more architectures like Transformer.

watts4speed commented 2 years ago

I'd like to second the request for support for transformers. This is a really important blocker for my projects. Great tool thanks for doing it.

VainF commented 2 years ago

Hi, FFN pruning for transformers has been supported in the latest version.