Open RonanKMcGovern opened 4 months ago
Can someone assist on this bug? Thanks
Hello I do have the same issue but from loading the adapter from a local directory
docker run -v /.cache/huggingface:/root/.cache/huggingface --env "HUGGING_FACE_HUB_TOKEN=..." -p 8000:8000 --ipc=host vllm/vllm-openai:latest --model meta-llama/Llama-2-7b-hf --enable-lora --lora-modules sql-my-test-lora=./FINE_TUNED_MODELS_TEXT_TO_SQL/checkpoint-500/
And I get the following error
lora = self._lora_model_cls.from_local_checkpoint(
File "/usr/local/lib/python3.10/dist-packages/vllm/lora/models.py", line 214, in from_local_checkpoint
with open(lora_config_path) as f:
FileNotFoundError: [Errno 2] No such file or directory: './FINE_TUNED_MODELS_TEXT_TO_SQL/checkpoint-500/adapter_config.json'
but inside the directory there is the adapter_config.json (notice I also display the models)
curl http://localhost:8892/v1/models
{"object":"list","data":[{"id":"meta-llama/Llama-2-7b-hf","object":"model","created":1716542398,"owned_by":"vllm","root":"meta-llama/Llama-2-7b-hf","parent":null,"permission":[{"id":"modelperm-9dcdb84d92de4bc7ba960fa844213025","object":"model_permission","created":1716542398,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]},{"id":"sql-my-test-lora","object":"model","created":1716542398,"owned_by":"vllm","root":"meta-llama/Llama-2-7b-hf","parent":null,"permission":[{"id":"modelperm-403c7eca17474cf795a88605cd75fa47","object":"model_permission","created":1716542398,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}
@RonanKMcGovern did you try to load adapters from a local directory?
Thanks
Your current environment
🐛 Describe the bug
I'm running the latest docker image and an openai style endpoint.
My command is:
I'm hitting the endpoint (on runpod) with:
The error is:
Note that the model does display correctly when I hit the /models endpoint:
Further notes: