Closed EdwardTheLegend closed 2 months ago
Just curious, what model/what environment variables do you have for your configuration of this endpoint?
No environment variables set apart from the model which was meta-llama/Meta-Llama-3-70B. I've since switched to a quantized version someone else made casperhansen/llama-3-70b-instruct-awq which works for me so this can be closed if you want.
Yes, unquantified 70B won't fit on 48GB
Hey I'm curious, what docker container did you use in the template? I can't get anything to work with LLama 3 so far
Which llama 3 size, which GPUs were you trying to use and in what quantity per worker?
Llama-3-8B(https://huggingface.co/meta-llama/Meta-Llama-3-8B). I was using the Nvidia 4090, 12 vCPU, 25GB RAM. 1 GPU/worker
Llama-3-8B(https://huggingface.co/meta-llama/Meta-Llama-3-8B). I was using the Nvidia 4090, 12 vCPU, 25GB RAM. 1 GPU/worker
1 x 24GB GPU should be fine for this, what do your environment variable settings look like, and since you're using a different configuration than 70B its not related to this issue, so you shouldn't be hijacking this issue with unrelated queries.
Have you tried launching it from the UI? Also, llama 3 is a gated model, so you need to gain access by filling out the form on the model's page on Hugging Face and provide your HuggingFace token when deploying. Alternatively, you can deploy an ungated reupload, such as NousResearch/Meta-Llama-3-8B-Instruct, which requires no token.
Im running a runpod serverless vLLM template with Llama 3 70B on 40GB GPU. One of the requests failed and I'm not completely sure what happened but the message asked me to open a github issue so I'll just leave it here in case it's of any help to any one.