Average validation loss across grad_accum_every

lucidrains / audiolm-pytorch

Implementation of AudioLM, a SOTA Language Modeling Approach to Audio Generation out of Google Research, in Pytorch

MIT License

2.32k stars 249 forks source link

Average validation loss across grad_accum_every #223

Closed LWprogramming closed 11 months ago

LWprogramming commented 11 months ago

Encountered this in a different training run where I was trying audio of longer lengths and ended up OOMing on large batch sizes, so I increased grad_accum_every to 32 and ended up with this graph:

Compare to after including this change:

lucidrains commented 11 months ago

accept! 💯