Closed LBYPatrick closed 1 year ago
In our DB-GPT framework, it is very simple to access the new model, and the specific access can be written according to the following tutorial.
Implement a ModelAdapter according to template. Add code at pilot/model/adapter.py, next is a demo of ChatGLM
class ChatGLMAdapater(BaseLLMAdaper):
"""LLM Adatpter for THUDM/chatglm-6b"""
def match(self, model_path: str):
return "chatglm" in model_path
def loader(self, model_path: str, from_pretrained_kwargs: dict):
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
if DEVICE != "cuda":
model = AutoModel.from_pretrained(
model_path, trust_remote_code=True, **from_pretrained_kwargs
).float()
return model, tokenizer
else:
model = (
AutoModel.from_pretrained(
model_path, trust_remote_code=True, **from_pretrained_kwargs
)
.half()
.cuda()
)
return model, tokenizer
2. Implement a ChatAdapter at pilot/server/chat_adapter.py
```python
class ChatGLMChatAdapter(BaseChatAdpter):
"""Model chat Adapter for ChatGLM"""
def match(self, model_path: str):
return "chatglm" in model_path
def get_generate_stream_func(self):
from pilot.model.llm_out.chatglm_llm import chatglm_generate_stream
return chatglm_generate_stream
Implement model generate function. pilot/model/llm_out/chatglm_llm.py
@torch.inference_mode()
def chatglm_generate_stream(
model, tokenizer, params, device, context_len=2048, stream_interval=2
):
"""Generate text using chatglm model's chat api"""
prompt = params["prompt"]
temperature = float(params.get("temperature", 1.0))
top_p = float(params.get("top_p", 1.0))
stop = params.get("stop", "###")
echo = params.get("echo", False)
generate_kwargs = {
"do_sample": True if temperature > 1e-5 else False,
"top_p": top_p,
"repetition_penalty": 1.0,
"logits_processor": None,
}
if temperature > 1e-5:
generate_kwargs["temperature"] = temperature
# TODO, Fix this
print(prompt)
messages = prompt.split(stop)
#
# # Add history conversation
hist = []
once_conversation = []
for message in messages[:-2]:
if len(message) <= 0:
continue
if "human:" in message:
once_conversation.append(message.split("human:")[1])
# elif "system:" in message:
# once_conversation.append(f"""###system:{message.split("system:")[1]} """)
elif "ai:" in message:
once_conversation.append(message.split("ai:")[1])
last_conversation = copy.deepcopy(once_conversation)
hist.append(last_conversation)
once_conversation = []
# else:
# once_conversation.append(f"""###system:{message} """)
try:
query = messages[-2].split("human:")[1]
except IndexError:
query = messages[-3].split("human:")[1]
print("Query Message: ", query)
# output = ""
# i = 0
for i, (response, new_hist) in enumerate(
model.stream_chat(tokenizer, query, hist, **generate_kwargs)
):
if echo:
output = query + " " + response
else:
output = response
yield output
yield output
just do this, you can use LLM whatever you want. Please try this, we really appreciate your contributions.
baichuan-7B is an open-source large-scale pre-trained model developed by Baichuan Intelligent Technology. Based on the Transformer architecture, it is a model with 7 billion parameters trained on approximately 1.2 trillion tokens. It supports both Chinese and English, with a context window length of 4096. It achieves the best performance of its size on standard Chinese and English authoritative benchmarks (C-EVAL/MMLU).
Here is a link to its model card on HuggingFace.
Would you mind putting this on the roadmap for support? Additionally, support
guanaco-33b-merged
is brokenas fuck. That is, we get something like this fromllmserver.py
when we do so:here's our system configuration as provided by
conda info
and here's our GPU:
Any effort is greatly appreciated
as it prevents my ass from getting wet.