csunny
commented
1 year ago In our DB-GPT framework, it is very simple to access the new model, and the specific access can be written according to the following tutorial.
-
Implement a ModelAdapter according to template. Add code at pilot/model/adapter.py, next is a demo of ChatGLM
class ChatGLMAdapater(BaseLLMAdaper): """LLM Adatpter for THUDM/chatglm-6b""" def match(self, model_path: str): return "chatglm" in model_path def loader(self, model_path: str, from_pretrained_kwargs: dict): tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) if DEVICE != "cuda": model = AutoModel.from_pretrained( model_path, trust_remote_code=True, **from_pretrained_kwargs ).float() return model, tokenizer else: model = ( AutoModel.from_pretrained( model_path, trust_remote_code=True, **from_pretrained_kwargs ) .half() .cuda() ) return model, tokenizer
2. Implement a ChatAdapter at pilot/server/chat_adapter.py
```python
class ChatGLMChatAdapter(BaseChatAdpter):
"""Model chat Adapter for ChatGLM"""
def match(self, model_path: str):
return "chatglm" in model_path
def get_generate_stream_func(self):
from pilot.model.llm_out.chatglm_llm import chatglm_generate_stream
return chatglm_generate_stream-
Implement model generate function. pilot/model/llm_out/chatglm_llm.py
@torch.inference_mode() def chatglm_generate_stream( model, tokenizer, params, device, context_len=2048, stream_interval=2 ): """Generate text using chatglm model's chat api""" prompt = params["prompt"] temperature = float(params.get("temperature", 1.0)) top_p = float(params.get("top_p", 1.0)) stop = params.get("stop", "###") echo = params.get("echo", False) generate_kwargs = { "do_sample": True if temperature > 1e-5 else False, "top_p": top_p, "repetition_penalty": 1.0, "logits_processor": None, } if temperature > 1e-5: generate_kwargs["temperature"] = temperature # TODO, Fix this print(prompt) messages = prompt.split(stop) # # # Add history conversation hist = [] once_conversation = [] for message in messages[:-2]: if len(message) <= 0: continue if "human:" in message: once_conversation.append(message.split("human:")[1]) # elif "system:" in message: # once_conversation.append(f"""###system:{message.split("system:")[1]} """) elif "ai:" in message: once_conversation.append(message.split("ai:")[1]) last_conversation = copy.deepcopy(once_conversation) hist.append(last_conversation) once_conversation = [] # else: # once_conversation.append(f"""###system:{message} """) try: query = messages[-2].split("human:")[1] except IndexError: query = messages[-3].split("human:")[1] print("Query Message: ", query) # output = "" # i = 0 for i, (response, new_hist) in enumerate( model.stream_chat(tokenizer, query, hist, **generate_kwargs) ): if echo: output = query + " " + response else: output = response yield output yield output
just do this, you can use LLM whatever you want. Please try this， we really appreciate your contributions.
baichuan-7B is an open-source large-scale pre-trained model developed by Baichuan Intelligent Technology. Based on the Transformer architecture, it is a model with 7 billion parameters trained on approximately 1.2 trillion tokens. It supports both Chinese and English, with a context window length of 4096. It achieves the best performance of its size on standard Chinese and English authoritative benchmarks (C-EVAL/MMLU).
Here is a link to its model card on HuggingFace.
Would you mind putting this on the roadmap for support? Additionally, support
guanaco-33b-mergedis brokenas fuck. That is, we get something like this fromllmserver.pywhen we do so:here's our system configuration as provided by
conda infoand here's our GPU:
Any effort is greatly appreciated
as it prevents my ass from getting wet.