search

QwenLM / qwen.cpp

C++ implementation of Qwen-LM
Other
538 stars 49 forks source link

How to use the python binding? #10

Open sunxichen opened 1 year ago

sunxichen commented 1 year ago

I successfully installed it from source, but I don't know how to use the chat and stream_chat interfaces, or how to load the model, etc. Are there any documents that I can refer to?

xwdreamer commented 1 year ago

how do you convert the model。

sunxichen commented 1 year ago

how do you convert the model。

Just follow the instruction in README. I converted the model in Intel Mac, not M1, and there is no error. I'm using transformers==4.33.2, maybe check your transformers package version?

simonJJJ commented 1 year ago

Now only supports tiktoken_cpp python bindings. I will release more python bindings within this week. Stay tuned.

quan2005 commented 12 months ago

Mark, I need Python bindings too.

simonJJJ commented 11 months ago

Python bindings are available now.

AppleJunJiang commented 11 months ago

Python bindings are available now.

怎么使用，没有在文档里面看到，目前只有tiktoken_cpp python bindings

yuebo commented 11 months ago

You can use below codes in python

from qwen_cpp import Pipeline                 
pipeline = Pipeline("/path_to_models/qwen7b-ggml.bin", "/path_to_tiktoken/Qwen-7B-Chat/qwen.tiktoken")

result1 = pipeline.chat(["Hello"])
print(result1)
result2 = pipeline.chat(["Hello"],stream=True)
for item in result2:
  print(item)
cailuyu commented 9 months ago

Seem python-bind did not support ”system”！ Anyway to support it? it should be easy to change the tokenizer to support

zzzcccxx commented 8 months ago

You can use below codes in python

from qwen_cpp import Pipeline                 
pipeline = Pipeline("/path_to_models/qwen7b-ggml.bin", "/path_to_tiktoken/Qwen-7B-Chat/qwen.tiktoken")

result1 = pipeline.chat(["Hello"])
print(result1)
result2 = pipeline.chat(["Hello"],stream=True)
for item in result2:
  print(item)

@yuebo 请问可以指定只在cpu上运行吗？