aarnphm
commented
11 months ago Can u try the following
OPENLLM_USE_LOCAL_LATEST=True openllm start llama —model-id huggyllama/llama-7b
?
Closed jeremythuon closed 10 months ago
aarnphm
commented
11 months ago Can u try the following
OPENLLM_USE_LOCAL_LATEST=True openllm start llama —model-id huggyllama/llama-7b
?
jeremythuon
commented
11 months ago Hi @aarnphm I try to add your env but it doesn't work Openllm try to find config.json
SError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like huggyllama/llama-7b is not the path to a directory containing a file named config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.I check with strace, openllm use the right path : stat("/home/outscale/bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16/model.yaml
(llm) [outscale@tata ~]$ ls -lah /home/outscale/bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16/ total 13G
drwxr-xr-x 2 outscale outscale 4.0K Oct 20 08:55 .
drwxr-xr-x 3 outscale outscale 68 Oct 19 12:46 ..
-rw-r--r-- 1 outscale outscale 42 Oct 19 12:37 added_tokens.json
-rw-r--r-- 1 outscale outscale 594 Oct 20 08:51 config.json
-rw-r--r-- 1 outscale outscale 137 Oct 19 12:37 generation_config.json
-rw-r--r-- 1 outscale outscale 11K Oct 19 12:37 LICENSE
-rw-r--r-- 1 outscale outscale 9.3G Oct 19 12:46 model-00001-of-00002.safetensors
-rw-r--r-- 1 outscale outscale 3.3G Oct 19 12:39 model-00002-of-00002.safetensors
-rw-r--r-- 1 outscale outscale 27K Oct 19 12:37 model.safetensors.index.json
-rw-r--r-- 1 outscale outscale 951 Oct 19 12:46 model.yaml
-rw-r--r-- 1 outscale outscale 27K Oct 19 12:37 pytorch_model.bin.index.json
-rw-r--r-- 1 outscale outscale 411 Oct 19 12:37 special_tokens_map.json
-rw-r--r-- 1 outscale outscale 700 Oct 19 12:37 tokenizer_config.json
-rw-r--r-- 1 outscale outscale 1.8M Oct 19 12:37 tokenizer.json
-rw-r--r-- 1 outscale outscale 489K Oct 19 12:37 tokenizer.modelDo you have an idea ?
aarnphm
commented
11 months ago Do you pass in HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 as well? this might not work. But I will def take a look thanks for reporting it.
jeremythuon
commented
11 months ago It's the same result with or whitout variable Do you think it comes from huggingface?
yingjie-han
commented
10 months ago Do you pass in
HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1as well? this might not work. But I will def take a look thanks for reporting it.
Hi, @aarnphm it is not working for me .Following is the log:
$ HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 openllm start baichuan --model-id /home/yingjie/openllm/baichuan2-13b --backend pt
Traceback (most recent call last):
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/openllm/serialisation/transformers/init.py", line 147, in get
model = bentoml.models.get(llm.tag)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/simpledi/init.py", line 139, in
return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/bentoml/models.py", line 45, in get
return _model_store.get(tag)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/bentoml/_internal/store.py", line 158, in get
raise NotFound(
bentoml.exceptions.NotFound: Model 'pt-baichuan2-13b:08c4d4d5d8625c6702b44beca2570febec83a4ae' is not found in BentoML store <osfs '/root/bentoml/models'>, you may need to run bentoml models pull first
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/openllm/cli/entrypoint.py", line 416, in import_command
_ref = openllm.serialisation.get(llm)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/openllm/serialisation/init.py", line 75, in caller
return getattr(importlib.import_module(f'.{serde}', name), fn)(llm, *args, **kwargs)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/openllm/serialisation/transformers/init.py", line 155, in get
raise openllm.exceptions.OpenLLMException(f'Failed while getting stored artefact (lookup for traceback):\n{err}') from err
openllm_core.exceptions.OpenLLMException: Failed while getting stored artefact (lookup for traceback):
Model 'pt-baichuan2-13b:08c4d4d5d8625c6702b44beca2570febec83a4ae' is not found in BentoML store <osfs '/root/bentoml/models'>, you may need to run bentoml models pull first
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/openllm/serialisation/transformers/init.py", line 147, in get
model = bentoml.models.get(llm.tag)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/simpledi/init.py", line 139, in
return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/bentoml/models.py", line 45, in get
return _model_store.get(tag)
File "/home/yingjie/openllm/v_openllm_bc/lib64/python3.8/site-packages/bentoml/_internal/store.py", line 158, in get
raise NotFound(
bentoml.exceptions.NotFound: Model 'pt-baichuan2-13b:08c4d4d5d8625c6702b44beca2570febec83a4ae' is not found in BentoML store <osfs '/root/bentoml/models'>, you may need to run bentoml models pull first
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/yingjie/openllm/v_openllm_bc/bin/openllm", line 8, in
aarnphm
commented
10 months ago will track this under #419
aarnphm
commented
10 months ago We have tested this and it seems to work for llama offline. Please try again. The baichuan model will tracked in a new thread instead.
Describe the bug
Hello,
I would like try openllm offline but I can't. For my test, I download huggyllama--llama-7b model with another computer with internet and I copy bento home to another computer. When I try to start with the command : HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 openllm start llama --model-id bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16
I have an error: Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.22s/it] Error: [bentoml-cli]
servefailed: Failed to generate a valid tag for llama with 'model_id=bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16/' (lookup to see its traceback): Can't load the configuration of 'bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'bentoml/models/pt-huggyllama--llama-7b/8416d3fefb0cb3ff5775a7b13c1692d10ff1aa16/' is the correct path to a directory containing a config.json fileCan you help me pls ?
Thanks
To reproduce
No response
Logs
No response
Environment
bentoml 1.1.7 openllm 0.3.9 openllm-client 0.3.9 openllm-core 0.3.9
System information (Optional)
No response