search

bentoml / OpenLLM

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
https://bentoml.com
Apache License 2.0
9.91k stars 630 forks source link

bug: openllm cannot start #743

Closed peijunzhang closed 10 months ago

peijunzhang commented 10 months ago

Describe the bug

When I start openllm by run "openllm start facebook/opt-1.3b --backend vllm", it stops starting.

To reproduce

  1. run 'openllm start facebook/opt-1.3b --backend vllm'
  2. error: ... ... File "/opt/miniconda3/envs/ai/lib/python3.11/site-packages/bentoml/_internal/utils/init.py", line 329, in validate_metadata metadata[key] = _validate_metadata_entry(val) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/ai/lib/python3.11/site-packages/bentoml/_internal/utils/init.py", line 372, in _validate_metadata_entry raise ValueError( ValueError: metadata entries must be basic python types like 'str', 'int', or 'complex', got 'function'

Logs

No response

Environment

transformers: 4.35.2 python: 3.11.4 Plantform: Ubuntu 20.04

System information (Optional)

No response

aarnphm commented 10 months ago
Screenshot 2023-11-30 at 09 44 21

This works perfectly fine here. Tested with python3.9

Can you send the output of pip freeze?

aarnphm commented 10 months ago
Screenshot 2023-11-30 at 09 51 41

Here testing on python3.11. Seems to works just fine

peijunzhang commented 10 months ago

@aarnphm

accelerate==0.23.0 aiofiles==23.2.1 aiohttp==3.8.5 aiosignal==1.3.1 altair==5.0.1 annotated-types==0.5.0 anyio==3.7.1 appdirs==1.4.4 argon2-cffi @ file:///opt/conda/conda-bld/argon2-cffi_1645000214183/work argon2-cffi-bindings @ file:///work/ci_py311/argon2-cffi-bindings_1676823553406/work asgiref==3.7.2 asttokens @ file:///opt/conda/conda-bld/asttokens_1646925590279/work async-timeout==4.0.3 attrs==23.1.0 azure-core==1.29.5 azure-storage-blob==12.19.0 backcall @ file:///home/ktietz/src/ci/backcall_1611930011877/work beautifulsoup4 @ file:///croot/beautifulsoup4-split_1681493039619/work bentoml==1.1.10 biopython==1.81 bitsandbytes==0.41.2.post2 bleach @ file:///opt/conda/conda-bld/bleach_1641577558959/work blinker==1.6.2 bokeh==3.2.2 build==0.10.0 cachetools==5.3.1 cattrs==23.1.2 certifi @ file:///croot/certifi_1690232220950/work/certifi cffi @ file:///work/ci_py311/cffi_1676822533496/work chardet @ file:///work/ci_py311/chardet_1676830276092/work charset-normalizer==3.2.0 circus==0.18.0 click==8.1.7 click-option-group==0.5.6 clickhouse-connect==0.6.9 cloudpickle==3.0.0 cmake==3.27.2 colorcet==3.0.1 coloredlogs==15.0.1 comm==0.1.4 contextlib2==21.6.0 contourpy==1.1.0 cpm-kernels==1.0.11 cryptography==41.0.7 cuda-python==12.3.0 cycler @ file:///tmp/build/80754af9/cycler_1637851556182/work dataclasses-json==0.5.14 datasets==2.15.0 debugpy==1.6.7.post1 decorator @ file:///opt/conda/conda-bld/decorator_1643638310831/work deepmerge==1.1.0 defusedxml @ file:///tmp/build/80754af9/defusedxml_1615228127516/work Deprecated==1.2.14 dill==0.3.7 distlib==0.3.7 distro==1.8.0 einops==0.7.0 entrypoints @ file:///work/ci_py311/entrypoints_1676823319002/work et-xmlfile==1.1.0 executing==1.2.0 faiss-cpu==1.7.4 fastapi==0.103.1 fastcore==1.5.29 fastjsonschema @ file:///work/ci_py311_2/python-fastjsonschema_1679340124475/work ffmpy==0.3.1 filelock==3.12.2 filetype==1.2.0 fonttools==4.25.0 frozenlist==1.4.0 fs==2.4.16 fschat==0.2.29 fsspec==2023.6.0 ghapi==1.0.4 gitdb==4.0.10 GitPython==3.1.32 google==3.0.0 google-api-core==2.11.1 google-api-python-client==2.97.0 google-auth==2.22.0 google-auth-httplib2==0.1.0 googleapis-common-protos==1.60.0 gradio==3.44.4 gradio_client==0.5.1 greenlet==2.0.2 h11==0.14.0 holoviews==1.17.1 httpcore==0.18.0 httplib2==0.22.0 httpx==0.25.0 huggingface-hub==0.19.4 humanfriendly==10.0 hvplot==0.8.4 idna @ file:///work/ci_py311/idna_1676822698822/work importlib-metadata==6.8.0 importlib-resources==6.1.0 inflection==0.5.1 ipyaggrid==0.4.1 ipycanvas==0.13.1 ipykernel @ file:///croot/ipykernel_1691121631942/work ipython==8.14.0 ipython-genutils @ file:///tmp/build/80754af9/ipython_genutils_1606773439826/work ipywidgets==8.1.0 ipywidgets-bokeh==1.5.0 isodate==0.6.1 jedi==0.19.0 Jinja2 @ file:///work/ci_py311/jinja2_1676823587943/work joblib==1.3.2 jsonlines==4.0.0 jsonschema @ file:///croot/jsonschema_1678983421637/work jupyter-server @ file:///work/ci_py311/jupyter_server_1676824947163/work jupyter_client==8.3.0 jupyter_core==5.3.1 jupyterlab-pygments @ file:///tmp/build/80754af9/jupyterlab_pygments_1601490720602/work jupyterlab-widgets==3.0.8 kiwisolver @ file:///work/ci_py311/kiwisolver_1676827230232/work lamini==1.0.3 lamini-configuration==0.8.3 langchain==0.0.271 langsmith==0.0.26 linkify-it-py==2.0.2 lit==16.0.6 lxml @ file:///croot/lxml_1679646450407/work lz4==4.3.2 Markdown==3.4.4 markdown-it-py==3.0.0 markdown2==2.4.10 MarkupSafe==2.1.3 marshmallow==3.20.1 matplotlib @ file:///croot/matplotlib-suite_1679593461707/work matplotlib-inline @ file:///work/ci_py311/matplotlib-inline_1676823841154/work mdit-py-plugins==0.4.0 mdurl==0.1.2 mistune @ file:///work/ci_py311/mistune_1676823698136/work mkl-fft==1.3.6 mkl-random @ file:///work/mkl/mkl_random_1682950433854/work mkl-service==2.4.0 mpmath==1.3.0 multidict==6.0.4 multiprocess==0.70.15 munkres==1.1.4 mypy-extensions==1.0.0 mysql-connector-python==8.1.0 nbclassic @ file:///croot/nbclassic_1681756170010/work nbclient @ file:///work/ci_py311/nbclient_1676823713954/work nbconvert @ file:///work/ci_py311/nbconvert_1676823854564/work nbformat @ file:///work/ci_py311/nbformat_1676823523130/work nest-asyncio==1.5.7 networkx==3.1 nh3==0.2.14 nltk==3.8.1 notebook @ file:///croot/notebook_1690984815942/work notebook_shim @ file:///work/ci_py311/notebook-shim_1676825053954/work numexpr==2.8.5 numpy @ file:///croot/numpy_and_numpy_base_1691091611330/work nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-cupti-cu11==11.7.101 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 nvidia-cufft-cu11==10.9.0.58 nvidia-curand-cu11==10.2.10.91 nvidia-cusolver-cu11==11.4.0.1 nvidia-cusparse-cu11==11.7.4.91 nvidia-ml-py==11.525.150 nvidia-nccl-cu11==2.14.3 nvidia-nvtx-cu11==11.7.91 openai==0.28.0 openllm==0.4.33 openllm-client==0.4.33 openllm-core==0.4.33 openpyxl==3.1.2 opentelemetry-api==1.20.0 opentelemetry-instrumentation==0.41b0 opentelemetry-instrumentation-aiohttp-client==0.41b0 opentelemetry-instrumentation-asgi==0.41b0 opentelemetry-sdk==1.20.0 opentelemetry-semantic-conventions==0.41b0 opentelemetry-util-http==0.41b0 optimum==1.14.1 orjson==3.9.7 packaging @ file:///croot/packaging_1678965309396/work pandas==2.0.3 pandocfilters @ file:///opt/conda/conda-bld/pandocfilters_1643405455980/work panel==1.2.1 param==1.13.0 parso @ file:///opt/conda/conda-bld/parso_1641458642106/work pathspec==0.11.2 peft==0.5.0 pexpect @ file:///tmp/build/80754af9/pexpect_1605563209008/work pickleshare @ file:///tmp/build/80754af9/pickleshare_1606932040724/work Pillow==9.4.0 pip-requirements-parser==32.0.1 pip-tools==7.3.0 platformdirs @ file:///croot/platformdirs_1692205439124/work ply==3.11 prometheus-client @ file:///work/ci_py311_2/prometheus_client_1679340520168/work prompt-toolkit==3.0.39 protobuf==4.21.12 psutil @ file:///work/ci_py311_2/psutil_1679337388738/work ptyprocess @ file:///tmp/build/80754af9/ptyprocess_1609355006118/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pure-eval @ file:///opt/conda/conda-bld/pure_eval_1646925070566/work pyarrow==13.0.0 pyarrow-hotfix==0.6 pyasn1==0.5.0 pyasn1-modules==0.3.0 pycparser @ file:///tmp/build/80754af9/pycparser_1636541352034/work pyct==0.5.0 pydantic==1.10.12 pydantic_core==2.6.3 pydeck==0.8.0 pydub==0.25.1 Pygments==2.16.1 Pympler==1.0.1 pyparsing @ file:///work/ci_py311/pyparsing_1677811559502/work pyperclip==1.8.2 pyproject_hooks==1.0.0 PyQt5-sip==12.11.0 pyrsistent @ file:///work/ci_py311/pyrsistent_1676823088752/work python-dateutil @ file:///tmp/build/80754af9/python-dateutil_1626374649649/work python-decouple==3.8 python-json-logger==2.0.7 python-multipart==0.0.6 pytz==2023.3 pytz-deprecation-shim==0.1.0.post0 pyviz_comms==3.0.0 PyYAML==6.0.1 pyzmq @ file:///work/ci_py311/pyzmq_1676918885689/work regex==2023.8.8 requests==2.31.0 rich==13.5.2 rise==5.7.1 rsa==4.9 safetensors==0.3.3 schema==0.7.5 scikit-learn==1.3.0 scipy==1.11.2 semantic-version==2.10.0 Send2Trash @ file:///tmp/build/80754af9/send2trash_1632406701022/work sentence-transformers==2.2.2 sentencepiece==0.1.99 shortuuid==1.0.11 simple-di==0.1.5 simplejson==3.19.1 sip @ file:///work/ci_py311/sip_1676825117084/work six @ file:///tmp/build/80754af9/six_1644875935023/work smmap==5.0.0 sniffio @ file:///work/ci_py311/sniffio_1676823742530/work soupsieve @ file:///croot/soupsieve_1680518478486/work SQLAlchemy==2.0.20 stack-data @ file:///opt/conda/conda-bld/stack_data_1646927590127/work starlette==0.27.0 stqdm==0.0.5 streamlit==1.27.2 streamlit-aggrid==0.3.4.post3 streamlit-ext==0.1.8 svgwrite==1.4.3 sympy==1.12 tenacity==8.2.3 terminado @ file:///work/ci_py311/terminado_1677696151350/work threadpoolctl==3.2.0 tiktoken==0.5.1 tinycss2 @ file:///work/ci_py311/tinycss2_1676823757641/work tokenizers==0.15.0 toml @ file:///tmp/build/80754af9/toml_1616166611790/work toolz==0.12.0 torch==2.0.1 torchvision==0.15.2 tornado==6.3.3 tqdm==4.66.1 traitlets==5.9.0 transformers==4.35.2 triton==2.0.0 typing-inspect==0.9.0 typing_extensions @ file:///croot/typing_extensions_1690297465030/work tzdata==2023.3 tzlocal==4.3.1 uc-micro-py==1.0.2 uritemplate==4.1.1 urllib3==1.26.16 uvicorn==0.23.2 validators==0.21.2 virtualenv==20.24.7 watchdog==3.0.0 watchfiles==0.21.0 wavedrom==2.0.3.post3 wcwidth @ file:///Users/ktietz/demo/mc3/conda-bld/wcwidth_1629357192024/work webencodings==0.5.1 websocket-client @ file:///work/ci_py311/websocket-client_1676824890004/work websockets==11.0.3 widgetsnbextension==4.0.8 wrapt==1.16.0 xlrd==2.0.1 XlsxWriter==3.1.2 xxhash==3.4.1 xyzservices==2023.7.0 yarl==1.9.2 zipp==3.16.2 zstandard==0.21.0

aarnphm commented 10 months ago

How do you install openllm again?

aarnphm commented 10 months ago

Doesn't seem like you have vllm available here?

peijunzhang commented 10 months ago

@aarnphm Thanks.

I fixed it by setting up a new python 11 conda evn. It runs fine now.

peijun