amyeroberts
commented
5 months ago Hi @levidehaan, thanks for raising this issue!
Could you provide a checkpoint for a public model which replicates this issue? Does this happen for any llama model? Do you observe this issue when not running through a flask app? i.e. does this work:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
import time
import logging
import os
print(torch.version.cuda)
modellocation = "/home/levi/projects/text-generation-webui/models/Upstage_SOLAR-10.7B-Instruct-v1.0"
tokenizer = AutoTokenizer.from_pretrained("/home/levi/projects/text-generation-webui/models/Upstage_SOLAR-10.7B-Instruct-v1.0")
model = AutoModelForCausalLM.from_pretrained(
modellocation,
device_map="auto"
)
start_time = time.time()
data = # fill in data here
conversation = data['messages']
prompt = tokenizer.apply_chat_template(
conversation,
tokenize=False,
add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, use_cache=True, max_length=4096)
outputs = model.generate(**inputs, use_cache=True, max_length=4096)
output_text = tokenizer.decode(outputs[0])
json_output = jsonify(
{'choices': [{'message': {'role': 'assistant', 'content': output_text}}]}
)
request_duration = time.time() - request.start_time
print(f"Request took {request_duration} seconds")
print(json_output)
Based on the error message, I'd be willing to bet it's an indexing issue with position_ids.
cc @gante as this covers rope, cuda and generate :)
BiEchi
gante
muellerzr
github-actions[bot]
System Info
Version: transformers-4.36.2
I have 2 RTX 3060's and i am able to run LLM's on One GPU but it wont work when i try to run them on 2 GPU's with the error:
Running Arch
It runs fine on single GPU, but when i try to run on multiple it does not like it. I have tried with oobabooga and vllm and none of them make a difference, it always fails. I have tried with many models of many sizes and types 7b 8x7b 13b 33b 34b, awq doesnt work, gptq doesnt work either.
my motherboard is a MSi MEG x570 with a AMD APU in it, and it has no option to disable ACS in the bios
GPU comms test came back good (i think ):
Let me know what else you need me to do/run/compile or download to help get this fixed.
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
my code:
Run it
python project.py
Expected behavior
it should run on 2 GPU's