search

OpenCSGs / llm-inference

llm-inference is a platform for publishing and managing llm inference, providing a wide range of out-of-the-box features for model deployment, such as UI, RESTful API, auto-scaling, computing resource management, monitoring, and more.
Apache License 2.0
69 stars 17 forks source link

vllm cannot address "runtime_env" #87

Closed depenglee1707 closed 6 months ago

depenglee1707 commented 6 months ago

for Qwen/Qwen-7B, we set runtime_env like this:

  initialization:
    runtime_env:
      pip: ["transformers_stream_generator", "tiktoken"]

but when start up, still get the exception:

ImportError: This modeling file requires the following packages that were not found in your environment: tiktoken. Run `pip install tiktoken`
SeanHH86 commented 6 months ago

Fixed.