unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
17.05k stars 1.17k forks source link

[FIXED] `wandb: WARNING The run_name` #1153

Open jhangmez opened 12 hours ago

jhangmez commented 12 hours ago

I've been training a modelo with llama 3.2 1b and this happened at this time, I was training 4 hours ago and this didn't happen. I tried to stop it and run again but then it doesn't work because I get this Error: wandb.init() before wandb.log()

image

I was traning this early morning and it doesn't show, but now do.

Is this a bug or what?

jhangmez commented 11 hours ago

I do more feedback when I train and cancel and then do it again

==((====))==  Unsloth - 2x faster free finetuning | Num GPUs = 1
   \\   /|    Num examples = 36,946 | Num Epochs = 1
O^O/ \_/ \    Batch size per device = 2 | Gradient Accumulation steps = 8
\        /    Total batch size = 16 | Total steps = 1,500
 "-____-"     Number of trainable parameters = 11,272,192
**** Unsloth: Please use our fixed gradient_accumulation_steps by updating transformers and Unsloth!
 [ 26/1500 06:30 < 6:39:44, 0.06 it/s, Epoch 0.01/1]
Step | Training Loss -- | --


==((====))== Unsloth - 2x faster free finetuning | Num GPUs = 1 \\ /| Num examples = 36,946 | Num Epochs = 1 O^O/ \_/ \ Batch size per device = 2 | Gradient Accumulation steps = 8 \ / Total batch size = 16 | Total steps = 1,500 "-____-" Number of trainable parameters = 11,272,192 **** Unsloth: Please use our fixed gradient_accumulation_steps by updating transformers and Unsloth! [ 26/1500 06:30 < 6:39:44, 0.06 it/s, Epoch 0.01/1] Step Training Loss --------------------------------------------------------------------------- Error Traceback (most recent call last) [](https://localhost:8080/#) in () ----> 1 trainer_stats = trainer.train() 7 frames [/usr/local/lib/python3.10/dist-packages/wandb/sdk/lib/preinit.py](https://localhost:8080/#) in preinit_wrapper(*args, **kwargs) 34 ) -> Callable: 35 def preinit_wrapper(*args: Any, **kwargs: Any) -> Any: ---> 36 raise wandb.Error(f"You must call wandb.init() before {name}()") 37 38 preinit_wrapper.__name__ = str(name) Error: You must call wandb.init() before wandb.log()
danielhanchen commented 11 hours ago

@jhangmez Coincidentally I just fixed it :) I updated all training notebooks - please edit the TrainingArguments part by adding report_to = "none". For example:


    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        ...
    ),

should be edited to:


    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        ...
        report_to = "none", # Use this for WandB etc
    ),