Roadmap - Githubissues

learning-at-home / lean_transformer

Memory-efficient transformer. Work in progress.

MIT License

19 stars 3 forks source link

Roadmap #6

Open justheuristic opened 2 years ago

justheuristic commented 2 years ago

[DONE] Version 0.3: it works decently enough in all use cases

extract from private debug-1 codebase
make base transformer general enough for ViT/SWIN/VGGTransformer integration
make into a pip-installable package

Version 0.4: it works

triton backend for sparsity (@justheuristic )
fit debug-1 into a colab-like enironment
- forward + backward, 11gb gpu, 16gb ram, full sequence length
- maybe triton backend can fix that automatically?
- chunked Attn / FFN forward (#4 )
customizable checkpoints (#4 )
some vision architectures (#2 )
basic inference for LeanGPT ( @xtinkt + @justheuristic )

Version 0.4-0.6:

memory-efficient attention kernel (@krunt + @xtinkt )
feature-rich inference (with prefices, various sampling & beam search variants)
pre-trained model zoo (convert sahaj2, calm, debug0 & 1, rubert)
demonstrable memory savings & performance comparison
efficient sota vision architectures

Finale:

If it works fine, consinder merging into HF transformers