Colab is not working

[yukiarimo]

Hello. Colab notebook: llama2-finetune-own-data is not working correctly!

In the code block:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "lm_head",
    ],
    bias="none",
    lora_dropout=0.05,  # Conventional
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, config)
print_trainable_parameters(model)

# Apply the accelerator. You can comment this out to remove the accelerator.
model = accelerator.prepare_model(model)

I'm getting the following error:

trainable params: 81108992 || all params: 3581521920 || trainable%: 2.264651559077991
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-09c3a8fb39e3> in <cell line: 25>()
     23 
     24 # Apply the accelerator. You can comment this out to remove the accelerator.
---> 25 model = accelerator.prepare_model(model)

/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py in prepare_model(self, model, device_placement, evaluation_mode)
   1325             current_device_index = current_device.index if isinstance(current_device, torch.device) else current_device
   1326 
-> 1327             if torch.device(current_device_index) != self.device:
   1328                 # if on the first device (GPU 0) we don't care
   1329                 if (self.device.index is not None) or (current_device_index != 0):

TypeError: device() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (torch.device device)
      didn't match because some of the arguments have invalid types: (!NoneType!)
 * (str type, int index)

Any ideas?

[bachhm-dev]

The same issue! Does anyone have a solution to fix it?

[bachhm-dev]

@yukiarimo Change step 7 to model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config, device_map="auto")

[yukiarimo]

@yukiarimo Change step 7 to model = AutoModelForCausalLM.from_pretrained(base_model_id, quantization_config=bnb_config, device_map="auto")

Thanks, I’ll try it today! By the way, do you know how to fine-tune Phi 3 vision with a text-only dataset? I tried, but it says 120 hours instead of 30 minutes as usual for LLaMA :(