search

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
MIT License
1.29k stars 80 forks source link

Instructions unclear #137

Open Anonym0us33 opened 1 year ago

Anonym0us33 commented 1 year ago

in windows replacing private.dockerfile with:

FROM hyperonym/basaran:0.15.3

# Copy model files
COPY vicuna128 D:\basaran\models\vicuna128

# Provide default environment variables
ENV MODEL="D:\basaran\models\vicuna128"
ENV MODEL_LOCAL_FILES_ONLY="true"
ENV SERVER_MODEL_NAME="vicuna128"

and doing docker build deployments\bundle\private.dockerfile gives a location error because it can't find the model under models/vicuna so i tried running:

basaran> docker run -p 80:80 -e MODEL="D:\basaran\models\vicuna128" --name fun hyperonym/basaran:0.15.3
>>
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/app/basaran/__main__.py", line 38, in <module>
    stream_model = load_model(
  File "/app/basaran/model.py", line 318, in load_model
    tokenizer = AutoTokenizer.from_pretrained(name_or_path, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 642, in from_pretrained
    tokenizer_config = get_tokenizer_config(pretrained_model_name_or_path, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 486, in get_tokenizer_config
    resolved_config_file = cached_file(
  File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 409, in cached_file
    resolved_file = hf_hub_download(
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 112, in _inner_fn
    validate_repo_id(arg_value)
  File "/usr/local/lib/python3.8/dist-packages/huggingface_hub/utils/_validators.py", line 166, in validate_repo_id
    raise HFValidationError(
huggingface_hub.utils._validators.HFValidationError: Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: 'D:\basaran\models\vicuna128'.

it runs normally and seems to default to a model called bigscience/bloomz-560m which runs on http://127.0.0.1/ is there any way to swap models after running the docker file?? or any way you could post the output of your terminal when running a private docker file in this way?

im not new to docker, but im also unemployed so have no one to ask about this.

peakji commented 1 year ago

Hi @Anonym0us33!

The COPY instruction copies files or directories from <src> to the filesystem of the container at <dest>. And you cannot copy files outside the build context when building a docker image.

Therefore, you should first change your working directory to D:\basaran\models\ and then use a Dockerfile like this:

FROM hyperonym/basaran:0.15.3

# Copy model files
COPY vicuna128 /models/vicuna128

# Provide default environment variables
ENV MODEL="/models/vicuna128"
ENV MODEL_LOCAL_FILES_ONLY="true"
ENV MODEL_HALF_PRECISION="true"
ENV SERVER_MODEL_NAME="vicuna128"