Open leonardxie opened 3 months ago
me too
I also tried to run a training script using Trainer class from Huggingface and I faced several issues and errors including:
devices
(lora weights created on cpu when the model is in gpu)dtype
(multiplication between float and bfloat16 when the model is on bfloat16)xLoRAConfig
class.
hi, thank you for your excellent work. Do you have any plans to share the training code? i want to reproduce the training but raises error as followers
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.