Using GPU in script?: NVIDIA GeForce RTX 4090 24GB
Using distributed or parallel set-up in script?:
Reproduction
I used own remote local machine via AnyDesk.
There happens some warning messages like “You’re using a LlamaTokenizerFast tokenizer. Please note that with a fast tokenizer, using the call method is faster than using a method to encode the text followed by a call to the pad method to get a padded encoding.” and “The input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in torch.float16.”
At first, I used windows environment, not linux. In the beginning, training was successful, with no crash, the model saved well after the training. After then, I added target module in lora config then training went well but crashed only after the training. Thought it was due to the individually compiled version of bitsandbytes, so I made wsl2 linux environment and tried to run the code but this time, training doesn't even start due to the following error: "Error out of memory at line 380 in file /mmfs1/gscratch/zlab/timdettmers/git/bitsandbytes/csrc/pythonInterface.c"
I didn't add any custom code I made, but only used hugging face library code for training.
I wonder that why does OOM happen even when shared GPU memory is avaliable about more than 100GB. Training only uses 3GB/128GB of shared GPU memory while using 23.5GB/24GB of dedicated GPU memory. Also, in the middle process of my project why did train went well while saving process didn't work properly. Is memory required more when saving the model?
By the fact that all of the process went well at the beginning, I think my hardware can support the training. Am I wrong??
System Info
transformers
version: 4.37.0.dev0Reproduction
Here is the code.
Expected behavior
I just want to know the answers of my questions in the reproduction section and way to solve the problem.