search

Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.21k stars 3.37k forks source link

Add truncated backpropagation through time (TBPTT) example #19985

Open svnv-svsv-jm opened 4 months ago

svnv-svsv-jm commented 4 months ago

📚 Documentation

It seems like there is not example anymore about how to deal with Truncated Backpropagation Through Time in the latest versions. Am I wrong?

cc @borda

awaelchli commented 3 months ago

Special support for TBPTT was removed in Lightning 2.0. The release notes here provide an example how you could implement it normally, see the section Truncated backpropagation through time (TBPTT).

Adding a TBTT example to the docs would be possible too. Contributions towards this would be welcomed.

svnv-svsv-jm commented 3 months ago

Yes, that would be good.

However, removing this means that I still have to load a huge time-window in memory, then loop over it in the training_step, which kind of defeats the purpose of TBPTT.

One would have to pass DataLoader's to the training_step in the batch argument for it to be less memory-intensive.