search

learning-at-home / lean_transformer

Memory-efficient transformer. Work in progress.
MIT License
19 stars 3 forks source link

Inference / LeanGPT.generate #5

Open justheuristic opened 2 years ago

justheuristic commented 2 years ago

This is a master discussion for memory-efficient inferencing, further notes will be added shortly

Current quest stage: add a dummy cache that is passed to all attention layers

justheuristic commented 2 years ago

ideally, this should be available as a .generate method in LeanGPTForPreTraining https://github.com/learning-at-home/lean_transformer/blob/main/lean_transformer/models/gpt.py#L184