Seeing something very strange when I try to train with pfeiffer+inv adapters. I'll try to replicate this after some more GPUs free up on my end, and no problem if no one can check this (probably hard to replicate and I'd bet it'll mysteriously go away at some point :P)
{'loss': 1604337.28, 'learning_rate': 0.0009997200000000002, 'epoch': 0.02} then loss = 0.0 by the next logging period, at 1k steps.
This is using this script (below) and the current main branch. It happens the same way both with commit f55ab013599088a35c87a880ba13a6d912e27ef4 and the current bloom branch.
Thanks Yong for trying to reproduce this bug! It didn't reproduce so seems it's a (hardware?) issue on my end causing NaNs / overflow/underflow in FP16.
Seeing something very strange when I try to train with pfeiffer+inv adapters. I'll try to replicate this after some more GPUs free up on my end, and no problem if no one can check this (probably hard to replicate and I'd bet it'll mysteriously go away at some point :P)
{'loss': 1604337.28, 'learning_rate': 0.0009997200000000002, 'epoch': 0.02}
then loss =0.0
by the next logging period, at 1k steps.This is using this script (below) and the current main branch. It happens the same way both with commit
f55ab013599088a35c87a880ba13a6d912e27ef4
and the currentbloom
branch.