search

TidalPaladin / combustion

Collection of helpers for deep learning with PyTorch
Apache License 2.0
4 stars 1 forks source link

Add Linformer layer #19

Open TidalPaladin opened 3 years ago

TidalPaladin commented 3 years ago

The Linformer layer looks very interesting w.r.t. managing computational complexity of transformers. Having such an implementation would prove useful for upcoming projects involving point cloud transformers or vision transformers. This could probably be accomplished by subclassing existing PyTorch attention layers and adding in the intermediate low rank transformation.

TidalPaladin commented 3 years ago

Would probably be better to go with a Performer implementation, which achieves linear complexity while making fewer assumptions about the attention matrices.