Incorrect path_or_model_id

[Sapessii]

Hi!

In the last few hours I'm getting this error while pulling any image from hugging face:

OSError: Incorrect path_or_model_id: ''. Please provide either the path to a local folder or the repo_id of a model on the Hub.

I am using the standard vllm template from runpod serverless.

Thank you!

[rafa-9]

same error @alpayariyak

[ashleykleynhans]

I am getting the same error @alpayariyak

[ashleykleynhans]

My model isn't pulled from Hugging Face though, it already exists on my Network Storage, I downloaded it manually and didn't use the vllm template in the "Explore" section to create my vllm endpoint, I created my template and endpoint manually.

[ashleykleynhans]

@Sapessii thats a different issue, please don't hijack this issue with unrelated issues.

[alpayariyak]

Are you all using the same model @Sapessii @rafa-9 @ashleykleynhans

[Sapessii]

I tried with

1. cognitivecomputations/dolphin-2.9.1-llama-3-8b
2. NousResearch/Hermes-2-Pro-Mistral-7B
3. mlabonne/NeuralHermes-2.5-Mistral-7B

[ashleykleynhans]

I am using meta-llama/Meta-Llama-3-70B

[alpayariyak]

Seems Huggingface made changes on their end for Tokenizers that breaks all Worker vLLM deployments, on it

[ashleykleynhans]

My cognitivecomputations/dolphin-2.9.1-llama-3-8b endpoint is actually working, @Sapessii

[ashleykleynhans]

Seems Huggingface made changes on their end for Tokenizers that breaks all Worker vLLM deployments, on it

Thanks, so what is the solution? Do they need to fix it or can we do something to fix it, or must you do something to fix it?

[ashleykleynhans]

Multiple people are reporting on Discord that their environment variables for their serverless endpoints just started getting lost. Could that not be the cause of this issue since the worker is heavily dependent on environment variables?

[ashleykleynhans]

Seems to be fine now that the issue with the environment variables getting lost is resolved, but I had to scale my workers down to zero and back up again so that the environment variables could be correctly applied and now I am no longer getting this error.

[alpayariyak]

Yes, I found that the environment variables set in the template are not read by the worker, we reverted the change on our end about half an hour ago! We're very sorry for any inconvenience, please contact us if it affected your production.