search

containers / ai-lab-recipes

Examples for building and running LLM services and applications locally with Podman
Apache License 2.0
111 stars 110 forks source link

HF_PRETRAINED_MODEL not working properly #692

Open axel7083 opened 3 months ago

axel7083 commented 3 months ago

Description

am not sure to perfectly understand the following

https://github.com/containers/ai-lab-recipes/blob/55610a8c90b6e72c6e9289513825112e6c5e99b1/model_servers/llamacpp_python/src/run.sh#L18-L20

But using the command

podman run -v /home/axel7083/Documents/models/:/models:Z -e MODEL_PATH=/models/functionary-small-v2.5.Q4_0.gguf -e HF_PRETRAINED_MODEL=meetkai/functionary-small-v2.5-GGUF -e CHAT_FORMAT=functionary-v2 -p 8000:8000 -e PORT=8000 ghcr.io/containers/llamacpp_python:latest

does not provide the proper value to --hf_pretrained_model_name_or_path.

Expected

The tokenizer_config.json should be downloaded at startup

image

Current

With the current, it does not download it

image

axel7083 commented 3 months ago

cc @Gregory-Pereira I think a \ is missing on the following line

https://github.com/containers/ai-lab-recipes/blob/55610a8c90b6e72c6e9289513825112e6c5e99b1/model_servers/llamacpp_python/src/run.sh#L19

but when adding it we got the following error

./run.sh: line 12: HF_PRETRAINED_MODEL: %=--hf_pretrained_model_name_or_path %: syntax error: operand expected (error token is "%=--hf_pretrained_model_name_or_path %")