Closed laura-burdick-sil closed 3 weeks ago
danielhanchen
commented
4 weeks ago @laura-burdick-sil Apologies on the issue - you can use Unsloth directly to load the adapter! Ie:
FastLanguageModel.from_pretrained("/root/test_adapter2")We auto detect LoRAs. The reason for this error is because Unsloth dynamically patches stuff, so using normal HF will break in the same script
laura-burdick-sil
commented
3 weeks ago I'm not sure that I fully understand. When I replace this code:
# Load base model and adapter
base_model = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3.2-3B-bnb-4bit")
tokenizer = AutoTokenizer.from_pretrained("unsloth/Llama-3.2-3B-bnb-4bit")
config = PeftConfig.from_pretrained("/root/test_adapter2")
model = PeftModel.from_pretrained(base_model, "/root/test_adapter2", config=config, device_map={"":0})
tokenizer = AutoTokenizer.from_pretrained("/root/test_adapter2")with this code:
model = FastLanguageModel.from_pretrained("/root/test_adapter2", device_map={"":0})
tokenizer = AutoTokenizer.from_pretrained("/root/test_adapter2")and try to train the model (using the same code as above to load the dataset and train), I get the following error:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[5], line 18
1 # Train on data
2 training_arguments = TrainingArguments(
3 per_device_train_batch_size = 2,
4 gradient_accumulation_steps = 4,
(...)
15 save_strategy="steps", # Save checkpoints based on steps.
16 )
---> 18 trainer = SFTTrainer(
19 model = model,
20 tokenizer = tokenizer,
21 train_dataset = dataset1,
22 dataset_text_field = "text",
23 max_seq_length = 1024,
24 dataset_num_proc = 2,
25 packing = False, # Can make training 5x faster for short sequences.
26 args = training_arguments,
27 )
29 torch.cuda.empty_cache()
30 trainer.train()
File [~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:101](https://vscode-remote+localhost-003a8898.vscode-resource.vscode-cdn.net/root/~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py:101), in _deprecate_arguments.._inner_deprecate_positional_args..inner_f(*args, **kwargs)
99 message += "\n\n" + custom_message
100 warnings.warn(message, FutureWarning)
--> 101 return f(*args, **kwargs)
File [~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:401](https://vscode-remote+localhost-003a8898.vscode-resource.vscode-cdn.net/root/~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/trl/trainer/sft_trainer.py:401), in SFTTrainer.__init__(self, model, args, data_collator, train_dataset, eval_dataset, tokenizer, model_init, compute_metrics, callbacks, optimizers, preprocess_logits_for_metrics, peft_config, dataset_text_field, packing, formatting_func, max_seq_length, infinite, num_of_sequences, chars_per_token, dataset_num_proc, dataset_batch_size, neftune_noise_alpha, model_init_kwargs, dataset_kwargs, eval_packing)
395 if tokenizer.padding_side is not None and tokenizer.padding_side != "right":
396 warnings.warn(
397 "You passed a tokenizer with `padding_side` not equal to `right` to the SFTTrainer. This might lead to some unexpected behaviour due to "
398 "overflow issues when training a model in half-precision. You might consider adding `tokenizer.padding_side = 'right'` to your code."
399 )
--> 401 super().__init__(
402 model=model,
403 args=args,
404 data_collator=data_collator,
405 train_dataset=train_dataset,
406 eval_dataset=eval_dataset,
407 tokenizer=tokenizer,
408 model_init=model_init,
409 compute_metrics=compute_metrics,
410 callbacks=callbacks,
411 optimizers=optimizers,
412 preprocess_logits_for_metrics=preprocess_logits_for_metrics,
413 )
415 # Add tags for models that have been loaded with the correct transformers version
416 if hasattr(self.model, "add_model_tags"):
File [~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py:165](https://vscode-remote+localhost-003a8898.vscode-resource.vscode-cdn.net/root/~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/utils/deprecation.py:165), in deprecate_kwarg..wrapper..wrapped_func(*args, **kwargs)
161 elif minimum_action in (Action.NOTIFY, Action.NOTIFY_ALWAYS):
162 # DeprecationWarning is ignored by default, so we use FutureWarning instead
163 warnings.warn(message, FutureWarning, stacklevel=2)
--> 165 return func(*args, **kwargs)
File [~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/trainer.py:587](https://vscode-remote+localhost-003a8898.vscode-resource.vscode-cdn.net/root/~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/trainer.py:587), in Trainer.__init__(self, model, args, data_collator, train_dataset, eval_dataset, processing_class, model_init, compute_loss_func, compute_metrics, callbacks, optimizers, optimizer_cls_and_kwargs, preprocess_logits_for_metrics)
582 # Bnb Quantized models doesn't support `.to` operation.
583 if (
584 self.place_model_on_device
585 and not getattr(model, "quantization_method", None) == QuantizationMethod.BITS_AND_BYTES
586 ):
--> 587 self._move_model_to_device(model, args.device)
589 # Force n_gpu to 1 to avoid DataParallel as MP will manage the GPUs
590 if self.is_model_parallel:
File [~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/trainer.py:860](https://vscode-remote+localhost-003a8898.vscode-resource.vscode-cdn.net/root/~/.clearml/venvs-builds/3.10/lib/python3.10/site-packages/transformers/trainer.py:860), in Trainer._move_model_to_device(self, model, device)
859 def _move_model_to_device(self, model, device):
--> 860 model = model.to(device)
861 # Moving a model to an XLA device disconnects the tied weights, so we have to retie them.
862 if self.args.parallel_mode == ParallelMode.TPU and hasattr(model, "tie_weights"):
AttributeError: 'tuple' object has no attribute 'to'Am I missing something here? Thank you!
Erland366
commented
3 weeks ago Ohhh, FastLanguageModel returning both model and tokenizer at the same time. So you need to unpack it :
model, tokenizer = FastLanguageModel.from_pretrained("/root/test_adapter2", max_seq_length=2048, load_in_4bit=True, dtype=None)Thank you! That worked!
In this code, I am loading a Lora adapter onto Llama 3.2 (3 billion), saving the adapter only, and then re-loading it to continue training. However, when I try to continue training, it errors out.
Here's the error that I'm getting:
Any ideas? Thank you!