search

TimDettmers / sparse_learning

Sparse learning library and sparse momentum resources.
MIT License
377 stars 45 forks source link

Does this library support NLP models such as Transformer #17

Open ghost opened 4 years ago

ghost commented 4 years ago

Hi, I am interested in this work. I want to try this algorithm to accelerate trainning procedure of NLP models. So I want to know if I can directly use this library on NLP models? Thanks!

TimDettmers commented 4 years ago

Yes, it should work without any problem. You can just follow the steps of wrapping the transformer into the Masking class and it should work just fine. What is happening in the background is that all weights in the module (and all its sub-modules) are multiplied with a binary mask before each forward pass.

If you apply this to transformers you should make sure though that you keep the layer norm parameters dense. You can achieve this by using the remove_type(torch.nn.LayerNorm) method fo the Masking class.

Let me know if you run into any problems.

nickyi1990 commented 1 year ago

Yes, it should work without any problem. You can just follow the steps of wrapping the transformer into the Masking class and it should work just fine. What is happening in the background is that all weights in the module (and all its sub-modules) are multiplied with a binary mask before each forward pass.

If you apply this to transformers you should make sure though that you keep the layer norm parameters dense. You can achieve this by using the remove_type(torch.nn.LayerNorm) method fo the Masking class.

Let me know if you run into any problems.

It's 2022 now, do you get any positive results?