search

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
https://bentoml.com
Apache License 2.0
9.7k stars 616 forks source link

bug: An exception occurred while instantiating runner 'llm-mistral-runner' #952

Closed billy-le closed 3 months ago

billy-le commented 4 months ago

Describe the bug

new install of openllm on python 3.12.2

running cmd: TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

To reproduce

  1. install pyenv: brew install pyenv
  2. add path to zsh: 
    echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
echo 'eval "$(pyenv init -)"' >> ~/.zshrc
  3. install pyenv-virtualenv: git clone https://github.com/pyenv/pyenv-virtualenv.git $(pyenv root)/plugins/pyenv-virtualenv
  4. echo 'eval "$(pyenv virtualenv-init -)"' >> ~/.zshrc
  5. install python 3.12.2: pyenv install 3.12.2
  6. set global: pyenv global 3.12.2
  7. install openllm: pip install openllm
  8. quickstart: TRUST_REMOTE_CODE=True openllm start mistralai/Mistral-7B-Instruct-v0.1

Logs

2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] An exception occurred while instantiating runner 'llm-mistral-runner', see details below:
2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] Traceback (most recent call last):
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
self._set_handle(LocalRunnerRef)
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
runner_handle = handle_class(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_runners.py", line 165, in __init__
self.llm, self.config, self.model, self.tokenizer = llm, llm.config, llm.model, llm.tokenizer
^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_llm.py", line 459, in model
model = openllm.serialisation.load_model(self, *self._model_decls, **self._model_attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/__init__.py", line 63, in caller
return getattr(importlib.import_module(f'.{serde}', 'openllm.serialisation'), fn)(llm, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/__init__.py", line 97, in load_model
auto_class = infer_autoclass_from_llm(llm, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/_helpers.py", line 33, in infer_autoclass_from_llm
raise ValueError(
ValueError: Invalid configuration for mistralai/Mistral-7B-Instruct-v0.1. ``trust_remote_code=True`` requires `transformers.PretrainedConfig` to contain a `auto_map` mapping

2024-04-14T16:54:13+0800 [ERROR] [runner:llm-mistral-runner:1] Traceback (most recent call last):
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/starlette/routing.py", line 732, in lifespan
async with self.lifespan_context(app) as maybe_state:
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/server/base_app.py", line 75, in lifespan
on_startup()
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 317, in init_local
raise e
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
self._set_handle(LocalRunnerRef)
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
runner_handle = handle_class(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
self._runnable = runner.runnable_class(**runner.runnable_init_params)  # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_runners.py", line 165, in __init__
self.llm, self.config, self.model, self.tokenizer = llm, llm.config, llm.model, llm.tokenizer
^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/_llm.py", line 459, in model
model = openllm.serialisation.load_model(self, *self._model_decls, **self._model_attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/__init__.py", line 63, in caller
return getattr(importlib.import_module(f'.{serde}', 'openllm.serialisation'), fn)(llm, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/__init__.py", line 97, in load_model
auto_class = infer_autoclass_from_llm(llm, config)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/billyle/.pyenv/versions/3.12.2/lib/python3.12/site-packages/openllm/serialisation/transformers/_helpers.py", line 33, in infer_autoclass_from_llm
raise ValueError(
ValueError: Invalid configuration for mistralai/Mistral-7B-Instruct-v0.1. ``trust_remote_code=True`` requires `transformers.PretrainedConfig` to contain a `auto_map` mapping

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.1.11 python: 3.12.2 platform: macOS-14.4.1-arm64-arm-64bit uid_gid: 501:20

pip_packages
``` accelerate==0.29.2 aiohttp==3.9.4 aiosignal==1.3.1 anyio==4.3.0 appdirs==1.4.4 asgiref==3.8.1 attrs==23.2.0 bentoml==1.1.11 bitsandbytes==0.41.3.post2 build==0.10.0 cattrs==23.1.2 certifi==2024.2.2 charset-normalizer==3.3.2 circus==0.18.0 click==8.1.7 click-option-group==0.5.6 cloudpickle==3.0.0 coloredlogs==15.0.1 contextlib2==21.6.0 datasets==2.18.0 deepmerge==1.1.1 Deprecated==1.2.14 dill==0.3.8 distlib==0.3.8 distro==1.9.0 einops==0.7.0 fastcore==1.5.29 filelock==3.13.4 filetype==1.2.0 frozenlist==1.4.1 fs==2.4.16 fsspec==2024.2.0 ghapi==1.0.5 h11==0.14.0 httpcore==1.0.5 httpx==0.27.0 huggingface-hub==0.22.2 humanfriendly==10.0 idna==3.7 importlib-metadata==6.11.0 inflection==0.5.1 Jinja2==3.1.3 markdown-it-py==3.0.0 MarkupSafe==2.1.5 mdurl==0.1.2 mpmath==1.3.0 multidict==6.0.5 multiprocess==0.70.16 mypy-extensions==1.0.0 networkx==3.3 numpy==1.26.4 nvidia-ml-py==11.525.150 openllm==0.4.44 openllm-client==0.4.44 openllm-core==0.4.44 opentelemetry-api==1.20.0 opentelemetry-instrumentation==0.41b0 opentelemetry-instrumentation-aiohttp-client==0.41b0 opentelemetry-instrumentation-asgi==0.41b0 opentelemetry-sdk==1.20.0 opentelemetry-semantic-conventions==0.41b0 opentelemetry-util-http==0.41b0 optimum==1.18.1 orjson==3.10.0 packaging==24.0 pandas==2.2.2 pathspec==0.12.1 pillow==10.3.0 pip-requirements-parser==32.0.1 pip-tools==7.3.0 platformdirs==4.2.0 prometheus_client==0.20.0 protobuf==5.26.1 psutil==5.9.8 pyarrow==15.0.2 pyarrow-hotfix==0.6 pydantic==1.10.15 Pygments==2.17.2 pyparsing==3.1.2 pyproject_hooks==1.0.0 python-dateutil==2.9.0.post0 python-json-logger==2.0.7 python-multipart==0.0.9 pytz==2024.1 PyYAML==6.0.1 pyzmq==25.1.2 regex==2023.12.25 requests==2.31.0 rich==13.7.1 safetensors==0.4.2 schema==0.7.5 scipy==1.13.0 sentencepiece==0.2.0 setuptools==69.5.1 simple-di==0.1.5 six==1.16.0 sniffio==1.3.1 starlette==0.37.2 sympy==1.12 tokenizers==0.15.2 torch==2.2.2 tornado==6.4 tqdm==4.66.2 transformers==4.39.3 typing_extensions==4.11.0 tzdata==2024.1 urllib3==2.2.1 uvicorn==0.29.0 virtualenv==20.25.1 watchfiles==0.21.0 wheel==0.43.0 wrapt==1.16.0 xxhash==3.4.1 yarl==1.9.4 zipp==3.18.1 ```

System information (Optional)

No response

yufeng1684 commented 4 months ago

you can run openllm without "TRUST_REMOTE_CODE=True ".

aarnphm commented 3 months ago

This seems like a users issue, mistral is already supported by transformers, so no need to use TRUST_REMOTE_CODE=True.