Open JhonDan1999 opened 4 months ago
Hi,
I want to use Qlora with Flan-t5
but I got this error
ValueError: Trying to set a tensor of shape torch.Size([4096, 4096]) in "weight" (which has shape torch.Size([8388608, 1])), this look incorrect.
here is my code
model_name = "google/flan-t5-xxl" bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type='nf4', bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, llm_int8_enable_fp32_cpu_offload=True ) model = AutoModelForSeq2SeqLM.from_pretrained( #for "google/flan-t5-xxl" model_name, quantization_config=bnb_config, device_map='auto', torch_dtype=torch.bfloat16, ) model.config.use_cache = False model.config.pretraining_tp = 1 model.eval() tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True) tokenizer.pad_token = tokenizer.eos_token
the error occur earn I call the generate here
# Generate model output using greedy decoding with torch.no_grad(): outputs = model.generate(**inputs, max_new_tokens=1, do_sample=False,top_p=None)
can you please help me with this
Hi,
I want to use Qlora with Flan-t5
but I got this error
here is my code
the error occur earn I call the generate here
can you please help me with this