Closed Blueman2 closed 3 months ago
@Blueman2 Apply only for linear layers.
False alarm, first step had nan gradients thanks to gradient rescaling for fp16.
@Blueman2 How did you solve that? I encountered this problem while training the Gemma-7B model. I applied GaLore only to the self-attention and MLP layers, just like the provided scripts. However, the loss value is too high and it has not converged after the first step.
I have converted the GaLore code to C++ (libtorch) and currently running into an issue where large layers (the initial embedding layer) is failing at SVD.
Layer is 31618x2624 I am running with full_matrices set to false already.
[W BatchLinearAlgebraLib.cpp:703] Warning: torch.linalg.svd: During SVD computation with the selected cusolver driver, batches 0 failed to converge. A more accurate method will be used to compute the SVD as a fallback. Check doc at https://pytorch.org/docs/stable/generated/torch.linalg.svd.html (function operator ()) [08:32:34.6859554] Projection failed: linalg.svd: The algorithm failed to converge because the input matrix is ill-conditioned or has too many repeated singular values (error code: 2623).
Is it a known issue and is there any workarounds for this?