Hello, I would like to discuss a problem I have with only the 4bit quantized gpt-j models (gpt-j-6B-alpaca-4bit-128g) with the help of the AutoGPTQ method when using the monkeypatch to train with lora
(this only happens with this type of models, models like vicuna or WizardLM work very well with monkeypatch)
ERROR:
Load Model ...
WARNING:The safetensors archive passed at models\bertin-gpt-j-6B-alpaca-4bit-128g\gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata.
Traceback (most recent call last):
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\server.py", line 932, in
shared.model, shared.tokenizer = load_model(shared.model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\models.py", line 153, in loadmodel
model, = load_model_llama(model_name)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\monkey_patch_gptq_lora.py", line 24, in load_model_llama
model, tokenizer = load_llama_model_4bit_low_ram(config_path, model_path, groupsize=shared.args.groupsize, is_v1_model=False)
File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\repositories\alpaca_lora_4bit\autograd_4bit.py", line 204, in load_llama_model_4bit_low_ram
model = accelerate.load_checkpoint_and_dispatch(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\big_modeling.py", line 479, in load_checkpoint_and_dispatch
load_checkpoint_in_model(
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 946, in load_checkpoint_in_model
set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype)
File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 135, in set_module_tensor_to_device
if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None:
AttributeError: 'NoneType' object has no attribute 'device'
pc: rtx 3060 (12GB) i7 10700 32 ram
Hello, I would like to discuss a problem I have with only the 4bit quantized gpt-j models (gpt-j-6B-alpaca-4bit-128g) with the help of the AutoGPTQ method when using the monkeypatch to train with lora
(this only happens with this type of models, models like vicuna or WizardLM work very well with monkeypatch)
ERROR: Load Model ... WARNING:The safetensors archive passed at models\bertin-gpt-j-6B-alpaca-4bit-128g\gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the save_pretrained method. Defaulting to 'pt' metadata. Traceback (most recent call last): File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\server.py", line 932, in shared.model, shared.tokenizer = load_model(shared.model_name) File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\models.py", line 153, in loadmodel model, = load_model_llama(model_name) File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\modules\monkey_patch_gptq_lora.py", line 24, in load_model_llama model, tokenizer = load_llama_model_4bit_low_ram(config_path, model_path, groupsize=shared.args.groupsize, is_v1_model=False) File "A:\LLMs_LOCAL\oobabooga_windows\text-generation-webui\repositories\alpaca_lora_4bit\autograd_4bit.py", line 204, in load_llama_model_4bit_low_ram model = accelerate.load_checkpoint_and_dispatch( File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\big_modeling.py", line 479, in load_checkpoint_and_dispatch load_checkpoint_in_model( File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 946, in load_checkpoint_in_model set_module_tensor_to_device(model, param_name, param_device, value=param, dtype=dtype) File "A:\LLMs_LOCAL\oobabooga_windows\installer_files\env\lib\site-packages\accelerate\utils\modeling.py", line 135, in set_module_tensor_to_device if old_value.device == torch.device("meta") and device not in ["meta", torch.device("meta")] and value is None: AttributeError: 'NoneType' object has no attribute 'device'
pc: rtx 3060 (12GB) i7 10700 32 ram