Open enori opened 10 months ago
try using serve run serve_configs/meta-llama--Llama-2-7b-chat-hf.yaml . I'll fix the docs to reflect that
Docs fixed here https://github.com/ray-project/ray-llm/pull/85
Thank you!
I used anyscale/ray-llm
as the docker image, as mentioned in another Issue
The result is changed. Ray Serve status is DEPLOY_FAILED, the message is The deployments ['VLLMDeployment:meta-llama--Llama-2-7b-chat-hf'] are UNHEALTHY.
Moreover, the container I deployed does not have a serve_configs
directory, so I had to create a new one.
I think it is better to add the serve_configs
directory in the docker image or mention the need for a copy of serve_configs
file in the documentation.
I will commit the changes to the documentation later from here as well.
I deployed on avitary head/worker pod on EKS cluster using KubeRay and tried to deploy the LLM model due to an error in the following command.
However, I couldn't deploy it.
I think the problem is compatibility with Python packages. Is there a requirements.txt file (e.g. pydantic) with the appropriate package versions?
The following is the commands I ran and the output.
Thank you for checking.