search

TomMakkink / transformers-for-rl

Investigating the use of Transformers as memory mechanisms in reinforcement learning.
MIT License
5 stars 0 forks source link

Linformer #5

Open TomMakkink opened 4 years ago

TomMakkink commented 4 years ago

Implement the linformer model from the following paper: https://arxiv.org/pdf/2006.04768.pdf The implementation will be based on the opensource implementation: https://github.com/tatp22/linformer-pytorch

It seems that the repo does a good job of abstracting the various components away to linear attention functions etc., so we should be able to adapt these functions in our codebase fairly easily. It also seems that they have reproduced some of the benchmarks on the linformer website, so that is a good sign. Sanity checks will still be required.

TomMakkink commented 4 years ago

Found an alternative implementation by the Authors: https://github.com/facebookresearch/pytext/pull/1407/files I have used this as a base.