search

opea-project / GenAIComps

GenAI components at micro-service level; GenAI service composer to create mega-service
Apache License 2.0
58 stars 120 forks source link

Instructions to build vllm:cpu is missing in llm comps #335

Open preethivenkatesh opened 2 months ago

preethivenkatesh commented 2 months ago

https://github.com/opea-project/GenAIComps/tree/main/comps/llms#1-start-microservice-with-python-option-1

docker run -it --name vllm_service -p 8008:80 -e HF_TOKEN=${HUGGINGFACEHUB_API_TOKEN} -v ./data:/data vllm:cpu /bin/bash -c "cd / && export VLLM_CPU_KVCACHE_SPACE=40 && python3 -m vllm.entrypoints.openai.api_server --model ${your_hf_llm_model} --port 80"

vllm:cpu is not in docker hub, instruction missing to build from source

preethivenkatesh commented 2 months ago

add the command bash build_docker_vllm.sh