search

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
MIT License
1.29k stars 81 forks source link

Getting error for model when using vicuna model #152

Closed djaffer closed 1 year ago

djaffer commented 1 year ago

2023-04-18 17:03:51 Traceback (most recent call last): 2023-04-18 17:03:51 File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main 2023-04-18 17:03:51 return _run_code(code, main_globals, None, 2023-04-18 17:03:51 File "/usr/lib/python3.8/runpy.py", line 87, in _run_code 2023-04-18 17:03:51 exec(code, run_globals) 2023-04-18 17:03:51 File "/app/basaran/main.py", line 38, in 2023-04-18 17:03:51 stream_model = load_model( 2023-04-18 17:03:51 File "/app/basaran/model.py", line 318, in load_model 2023-04-18 17:03:51 tokenizer = AutoTokenizer.from_pretrained(name_or_path, kwargs) 2023-04-18 17:03:51 File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/tokenization_auto.py", line 657, in from_pretrained 2023-04-18 17:03:51 config = AutoConfig.from_pretrained( 2023-04-18 17:03:51 File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/configuration_auto.py", line 916, in from_pretrained 2023-04-18 17:03:51 config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, kwargs) 2023-04-18 17:03:51 File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 573, in get_config_dict 2023-04-18 17:03:51 config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) 2023-04-18 17:03:51 File "/usr/local/lib/python3.8/dist-packages/transformers/configuration_utils.py", line 628, in _get_config_dict 2023-04-18 17:03:51 resolved_config_file = cached_file( 2023-04-18 17:03:51 File "/usr/local/lib/python3.8/dist-packages/transformers/utils/hub.py", line 380, in cached_file 2023-04-18 17:03:51 raise EnvironmentError( 2023-04-18 17:03:51 OSError: /models/vicuna does not appear to have a file named config.json. Checkout 'https://huggingface.co//models/vicuna/None' for available files.

Maybe the documentation can improve on running a custom model. It is pretty vague right now.

FROM hyperonym/basaran:0.15.3

# Copy model files
COPY ./model /models/vicuna

# Provide default environment variables
ENV MODEL="/models/vicuna"
ENV MODEL_LOCAL_FILES_ONLY="true"
ENV MODEL_HALF_PRECISION="true"
ENV SERVER_MODEL_NAME="vicuna"
fardeon commented 1 year ago

It looks like the config.json file was not found in /models/vicuna. You can enter the container to check if the file has been copied correctly. This may be due to the different working directory when running docker build.

djaffer commented 1 year ago

it quits before it starts.

fardeon commented 1 year ago

it quits before it starts.

You may temporarily remove ENV MODEL="/models/vicuna" to let it download and start with the default model, meanwhile bash into the container to see if the files were correctly copied.

fardeon commented 1 year ago

Vicuna is now tested and is working properly (https://github.com/hyperonym/basaran/issues/160). This issue seems to be a configuration problem within the model repository.

peakji commented 1 year ago

I guess it is now safe to close this issue as vicuna has been confirmed to work with Basaran (https://github.com/hyperonym/basaran/issues/160, https://github.com/hyperonym/basaran/issues/180).

For choices of working vicuna models, please refer to https://github.com/hyperonym/basaran/issues/160#issuecomment-1518394030.