I am trying to deploy GPTQ Llama-2 model using OpenLLM. Did not use any configurations, but simply ran the line from the readme documentation.
When tracing the error, it seems like the configurations class in AutoGPTQ is the issue. Not sure if there's a workaround or a fix that can be done in this repo but I'll raise this bug in the other repo as well.
2023-08-31T04:27:26+0000 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "_service:svc" can be accessed at http://localhost:3000/metrics.
2023-08-31T04:27:26+0000 [INFO] [cli] Starting production HTTP BentoServer from "_service:svc" listening on http://0.0.0.0:3000 (Press CTRL+C to quit)
2023-08-31T04:27:34+0000 [ERROR] [runner:llm-llama-runner:1] An exception occurred while instantiating runner 'llm-llama-runner', see details below:
2023-08-31T04:27:34+0000 [ERROR] [runner:llm-llama-runner:1] Traceback (most recent call last):
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 307, in init_local
self._set_handle(LocalRunnerRef)
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/bentoml/_internal/runner/runner.py", line 150, in _set_handle
runner_handle = handle_class(self, *args, **kwargs)
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/bentoml/_internal/runner/runner_handle/local.py", line 27, in __init__
self._runnable = runner.runnable_class(**runner.runnable_init_params) # type: ignore
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/openllm/_llm.py", line 1143, in __init__
if not self.model: raise RuntimeError('Failed to load the model correctly (See traceback above)')
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/openllm/_llm.py", line 795, in model
model = self.load_model(*self._model_decls, **self._model_attrs)
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/openllm/_llm.py", line 265, in wrapper
return f(self, *(*model_decls, *decls), **{**model_attrs, **attrs})
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/openllm/serialisation/__init__.py", line 76, in caller
return getattr(importlib.import_module(f'.{llm.runtime}', __name__), fn)(llm, *args, **kwargs)
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/openllm/serialisation/transformers/__init__.py", line 177, in load_model
model = infer_autoclass_from_llm(llm, config).from_pretrained(
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 516, in from_pretrained
return model_class.from_pretrained(
File "/home/ec2-user/SageMaker/llm_server/.env_openllm/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2470, in from_pretrained
quantization_method_from_config = config.quantization_config.get(
AttributeError: 'BaseQuantizeConfig' object has no attribute 'get'
Describe the bug
I am trying to deploy GPTQ Llama-2 model using OpenLLM. Did not use any configurations, but simply ran the line from the readme documentation.
When tracing the error, it seems like the configurations class in AutoGPTQ is the issue. Not sure if there's a workaround or a fix that can be done in this repo but I'll raise this bug in the other repo as well.
To reproduce
Logs
Environment
Environment variable
System information
bentoml
: 1.1.4python
: 3.10.8platform
: Linux-5.10.184-175.731.amzn2.x86_64-x86_64-with-glibc2.26uid_gid
: 1000:1000conda
: 22.11.1in_conda_env
: Trueconda_packages
```yaml name: JupyterSystemEnv channels: - https://aws-ml-conda-preview.s3.us-west-2.amazonaws.com - nvidia/label/cuda-11.8.0 - pytorch - conda-forge - intel dependencies: - _libgcc_mutex=0.1=conda_forge - _openmp_mutex=4.5=2_gnu - aiobotocore=2.5.4=pyhd8ed1ab_0 - aiohttp=3.8.5=py310h2372a71_0 - aioitertools=0.11.0=pyhd8ed1ab_0 - aiosignal=1.3.1=pyhd8ed1ab_0 - alsa-lib=1.2.3.2=h166bdaf_0 - anyio=3.7.1=pyhd8ed1ab_0 - argon2-cffi=23.1.0=pyhd8ed1ab_0 - argon2-cffi-bindings=21.2.0=py310h5764c6d_3 - arrow=1.2.3=pyhd8ed1ab_0 - asttokens=2.2.1=pyhd8ed1ab_0 - async-timeout=4.0.3=pyhd8ed1ab_0 - atk-1.0=2.38.0=hd4edc92_1 - attrs=23.1.0=pyh71513ae_1 - autovizwidget=0.20.5=pyh1a96a4e_0 - backcall=0.2.0=pyh9f0ad1d_0 - backports=1.0=pyhd8ed1ab_3 - backports.functools_lru_cache=1.6.5=pyhd8ed1ab_0 - beautifulsoup4=4.12.2=pyha770c72_0 - bleach=6.0.0=pyhd8ed1ab_0 - brotlipy=0.7.0=py310h5764c6d_1005 - bzip2=1.0.8=h7f98852_4 - ca-certificates=2023.7.22=hbcca054_0 - cached-property=1.5.2=hd8ed1ab_1 - cached_property=1.5.2=pyha770c72_1 - cairo=1.16.0=ha12eb4b_1010 - certifi=2023.7.22=pyhd8ed1ab_0 - cffi=1.15.1=py310h255011f_3 - charset-normalizer=3.2.0=pyhd8ed1ab_0 - comm=0.1.4=pyhd8ed1ab_0 - cryptography=39.0.0=py310h65dfdc0_0 - dbus=1.13.6=h5008d03_3 - debugpy=1.6.8=py310hc6cd4ac_0 - decorator=5.1.1=pyhd8ed1ab_0 - defusedxml=0.7.1=pyhd8ed1ab_0 - entrypoints=0.4=pyhd8ed1ab_0 - exceptiongroup=1.1.3=pyhd8ed1ab_0 - executing=1.2.0=pyhd8ed1ab_0 - expat=2.5.0=hcb278e6_1 - font-ttf-dejavu-sans-mono=2.37=hab24e00_0 - font-ttf-inconsolata=3.000=h77eed37_0 - font-ttf-source-code-pro=2.038=h77eed37_0 - font-ttf-ubuntu=0.83=hab24e00_0 - fontconfig=2.14.2=h14ed4e7_0 - fonts-conda-ecosystem=1=0 - fonts-conda-forge=1=0 - fqdn=1.5.1=pyhd8ed1ab_0 - freetype=2.12.1=hca18f0e_1 - fribidi=1.0.10=h36c2ea0_0 - frozenlist=1.4.0=py310h2372a71_0 - fsspec=2021.7.0=pyhd8ed1ab_0 - gdk-pixbuf=2.42.8=hff1cb4f_1 - gettext=0.21.1=h27087fc_0 - giflib=5.2.1=h0b41bf4_3 - glib=2.76.4=hfc55251_0 - glib-tools=2.76.4=hfc55251_0 - gmp=6.2.1=h58526e2_0 - graphite2=1.3.13=h58526e2_1001 - graphviz=4.0.0=h5abf519_0 - gst-plugins-base=1.20.2=hcf0ee16_0 - gstreamer=1.20.3=hd4edc92_2 - gtk2=2.24.33=h90689f9_2 - gts=0.7.6=h977cf35_4 - harfbuzz=4.2.0=h40b6f09_0 - hdijupyterutils=0.20.5=pyh1a96a4e_0 - icu=69.1=h9c3ff4c_0 - idna=3.4=pyhd8ed1ab_0 - importlib_resources=6.0.1=pyhd8ed1ab_0 - ipython=8.14.0=pyh41d4057_0 - ipython_genutils=0.2.0=py_1 - ipywidgets=8.1.0=pyhd8ed1ab_0 - isoduration=20.11.0=pyhd8ed1ab_0 - jinja2=3.1.2=pyhd8ed1ab_1 - jmespath=1.0.1=pyhd8ed1ab_0 - jpeg=9e=h0b41bf4_3 - jsonpointer=2.0=py_0 - jsonschema=4.19.0=pyhd8ed1ab_1 - jsonschema-specifications=2023.7.1=pyhd8ed1ab_0 - jsonschema-with-format-nongpl=4.19.0=pyhd8ed1ab_1 - jupyter=1.0.0=py310hff52083_8 - jupyter_console=6.6.3=pyhd8ed1ab_0 - jupyter_core=5.3.1=py310hff52083_0 - jupyter_events=0.7.0=pyhd8ed1ab_2 - jupyter_server=2.7.1=pyhd8ed1ab_0 - jupyter_server_terminals=0.4.4=pyhd8ed1ab_1 - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0 - jupyterlab_widgets=3.0.8=pyhd8ed1ab_0 - keyutils=1.6.1=h166bdaf_0 - krb5=1.20.1=hf9c8cef_0 - lcms2=2.14=h6ed2654_0 - ld_impl_linux-64=2.40=h41732ed_0 - lerc=4.0.0=h27087fc_0 - libabseil=20230125.3=cxx17_h59595ed_0 - libblas=3.9.0=17_linux64_openblas - libcblas=3.9.0=17_linux64_openblas - libclang=13.0.1=default_h7634d5b_3 - libcups=2.3.3=h36d4200_3 - libdeflate=1.14=h166bdaf_0 - libedit=3.1.20191231=he28a2e2_2 - libevent=2.1.10=h9b69904_4 - libexpat=2.5.0=hcb278e6_1 - libffi=3.4.2=h7f98852_5 - libgcc-ng=13.1.0=he5830b7_0 - libgd=2.3.3=h283352f_2 - libgfortran-ng=13.1.0=h69a702a_0 - libgfortran5=13.1.0=h15d22d2_0 - libglib=2.76.4=hebfc3b9_0 - libgomp=13.1.0=he5830b7_0 - libiconv=1.17=h166bdaf_0 - liblapack=3.9.0=17_linux64_openblas - libllvm13=13.0.1=hf817b99_2 - libnsl=2.0.0=h7f98852_0 - libogg=1.3.4=h7f98852_1 - libopenblas=0.3.23=pthreads_h80387f5_0 - libopus=1.3.1=h7f98852_1 - libpng=1.6.39=h753d276_0 - libpq=14.5=h2baec63_5 - libprotobuf=4.23.4=hd1fb520_1 - librsvg=2.54.4=h7abd40a_0 - libsodium=1.0.18=h36c2ea0_1 - libsqlite=3.42.0=h2797004_0 - libstdcxx-ng=13.1.0=hfd8a6a1_0 - libtiff=4.4.0=h82bc61c_5 - libtool=2.4.7=h27087fc_0 - libuuid=2.38.1=h0b41bf4_0 - libuv=1.43.0=h7f98852_0 - libvorbis=1.3.7=h9c3ff4c_0 - libwebp=1.2.4=h522a892_0 - libwebp-base=1.2.4=h166bdaf_0 - libxcb=1.13=h7f98852_1004 - libxkbcommon=1.0.3=he3ba5ed_0 - libxml2=2.9.14=haae042b_4 - libzlib=1.2.13=hd590300_5 - markupsafe=2.1.3=py310h2372a71_0 - matplotlib-inline=0.1.6=pyhd8ed1ab_0 - mistune=2.0.5=pyhd8ed1ab_0 - mock=5.1.0=pyhd8ed1ab_0 - multidict=6.0.4=py310h1fa729e_0 - mysql-common=8.0.32=h14678bc_0 - mysql-libs=8.0.32=h54cf53e_0 - nb_conda=2.2.1=unix_6 - nb_conda_kernels=2.3.1=py310hff52083_2 - nbclassic=0.5.1=pyhd8ed1ab_0 - nbclient=0.8.0=pyhd8ed1ab_0 - nbconvert=7.2.9=pyhd8ed1ab_0 - nbconvert-core=7.2.9=pyhd8ed1ab_0 - nbconvert-pandoc=7.2.9=pyhd8ed1ab_0 - nbformat=5.9.2=pyhd8ed1ab_0 - ncurses=6.4=hcb278e6_0 - nest-asyncio=1.5.6=pyhd8ed1ab_0 - nodejs=14.18.3=h8ca31f7_2 - nose=1.3.7=py_1006 - notebook=6.5.5=pyha770c72_0 - notebook-shim=0.2.3=pyhd8ed1ab_0 - nspr=4.35=h27087fc_0 - nss=3.92=h1d7d5a4_0 - numpy=1.25.2=py310ha4c1d20_0 - openjdk=11.0.1=h516909a_1016 - openssl=1.1.1v=hd590300_0 - overrides=7.4.0=pyhd8ed1ab_0 - packaging=23.1=pyhd8ed1ab_0 - pandas=2.0.3=py310h7cbd5c2_1 - pandoc=3.1.3=h32600fe_0 - pandocfilters=1.5.0=pyhd8ed1ab_0 - pango=1.50.7=hbd2fdc8_0 - parso=0.8.3=pyhd8ed1ab_0 - pcre2=10.40=hc3806b6_0 - pexpect=4.8.0=pyh1a96a4e_2 - pickleshare=0.7.5=py_1003 - pip=23.2.1=pyhd8ed1ab_0 - pixman=0.40.0=h36c2ea0_0 - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_0 - platformdirs=3.10.0=pyhd8ed1ab_0 - plotly=5.16.1=pyhd8ed1ab_0 - prometheus_client=0.17.1=pyhd8ed1ab_0 - prompt-toolkit=3.0.39=pyha770c72_0 - prompt_toolkit=3.0.39=hd8ed1ab_0 - protobuf=4.23.4=py310hb875b13_0 - psutil=5.9.5=py310h1fa729e_0 - psycopg2=2.9.3=py310h5764c6d_1 - pthread-stubs=0.4=h36c2ea0_1001 - ptyprocess=0.7.0=pyhd3deb0d_0 - pure_eval=0.2.2=pyhd8ed1ab_0 - pycparser=2.21=pyhd8ed1ab_0 - pygments=2.16.1=pyhd8ed1ab_0 - pykerberos=1.2.4=py310hd1ceca7_3 - pyopenssl=23.2.0=pyhd8ed1ab_1 - pyqt=5.12.3=py310hff52083_8 - pyqt-impl=5.12.3=py310h1f8e252_8 - pyqt5-sip=4.19.18=py310h122e73d_8 - pyqtchart=5.12=py310hfcd6d55_8 - pyqtwebengine=5.12.1=py310hfcd6d55_8 - pysocks=1.7.1=pyha2e5f31_6 - python=3.10.8=h257c98d_0_cpython - python-dateutil=2.8.2=pyhd8ed1ab_0 - python-fastjsonschema=2.18.0=pyhd8ed1ab_0 - python-json-logger=2.0.7=pyhd8ed1ab_0 - python-tzdata=2023.3=pyhd8ed1ab_0 - python_abi=3.10=3_cp310 - pytz=2023.3=pyhd8ed1ab_0 - pyyaml=6.0.1=py310h2372a71_0 - pyzmq=24.0.1=py310h330234f_1 - qt=5.12.9=h1304e3e_6 - qtconsole=5.4.3=pyhd8ed1ab_0 - qtconsole-base=5.4.3=pyha770c72_0 - qtpy=2.3.1=pyhd8ed1ab_0 - readline=8.2=h8228510_1 - referencing=0.30.2=pyhd8ed1ab_0 - requests=2.31.0=pyhd8ed1ab_0 - requests-kerberos=0.12.0=py310hff52083_4 - rfc3339-validator=0.1.4=pyhd8ed1ab_0 - rfc3986-validator=0.1.1=pyh9f0ad1d_0 - rpds-py=0.9.2=py310hcb5633a_0 - s3fs=2021.7.0=pyhd8ed1ab_0 - s3transfer=0.6.2=pyhd8ed1ab_0 - send2trash=1.8.2=pyh41d4057_0 - setuptools=68.1.2=pyhd8ed1ab_0 - six=1.16.0=pyh6c4a22f_0 - sniffio=1.3.0=pyhd8ed1ab_0 - soupsieve=2.3.2.post1=pyhd8ed1ab_0 - sparkmagic=0.20.5=pyhd8ed1ab_0 - sqlite=3.42.0=h2c6b66d_0 - stack_data=0.6.2=pyhd8ed1ab_0 - tenacity=8.2.3=pyhd8ed1ab_0 - terminado=0.17.1=pyh41d4057_0 - tinycss2=1.2.1=pyhd8ed1ab_0 - tk=8.6.12=h27826a3_0 - tornado=6.3.3=py310h2372a71_0 - traitlets=5.9.0=pyhd8ed1ab_0 - typescript=3.6.4=0 - typing-extensions=4.7.1=hd8ed1ab_0 - typing_extensions=4.7.1=pyha770c72_0 - typing_utils=0.1.0=pyhd8ed1ab_0 - tzdata=2023c=h71feb2d_0 - uri-template=1.3.0=pyhd8ed1ab_0 - wcwidth=0.2.6=pyhd8ed1ab_0 - webcolors=1.13=pyhd8ed1ab_0 - webencodings=0.5.1=py_1 - websocket-client=1.6.2=pyhd8ed1ab_0 - wheel=0.41.2=pyhd8ed1ab_0 - widgetsnbextension=4.0.8=pyhd8ed1ab_0 - wrapt=1.15.0=py310h1fa729e_0 - xorg-fixesproto=5.0=h7f98852_1002 - xorg-inputproto=2.3.2=h7f98852_1002 - xorg-kbproto=1.0.7=h7f98852_1002 - xorg-libice=1.1.1=hd590300_0 - xorg-libsm=1.2.4=h7391055_0 - xorg-libx11=1.8.4=h0b41bf4_0 - xorg-libxau=1.0.11=hd590300_0 - xorg-libxdmcp=1.1.3=h7f98852_0 - xorg-libxext=1.3.4=h0b41bf4_2 - xorg-libxfixes=5.0.3=h7f98852_1004 - xorg-libxi=1.7.10=h7f98852_0 - xorg-libxrender=0.9.10=h7f98852_1003 - xorg-libxtst=1.2.3=h7f98852_1002 - xorg-recordproto=1.14.2=h7f98852_1002 - xorg-renderproto=0.11.1=h7f98852_1002 - xorg-xextproto=7.3.0=h0b41bf4_1003 - xorg-xproto=7.0.31=h7f98852_1007 - xz=5.2.6=h166bdaf_0 - yaml=0.2.5=h7f98852_2 - yarl=1.9.2=py310h2372a71_0 - zeromq=4.3.4=h9c3ff4c_1 - zipp=3.16.2=pyhd8ed1ab_0 - zlib=1.2.13=hd590300_5 - zstd=1.5.2=hfc55251_7 - pip: - aiofiles==22.1.0 - aiosqlite==0.19.0 - astroid==2.15.6 - autopep8==2.0.2 - awscli==1.29.33 - babel==2.12.1 - boto3==1.28.33 - botocore==1.31.33 - cloudpickle==2.2.1 - colorama==0.4.4 - contextlib2==21.6.0 - dill==0.3.7 - docker==6.1.3 - docstring-to-markdown==0.12 - docutils==0.16 - environment-kernels==1.2.0 - gitdb==4.0.10 - gitpython==3.1.32 - google-pasta==0.2.0 - importlib-metadata==6.8.0 - ipykernel==5.5.6 - isort==5.12.0 - jedi==0.18.2 - json5==0.9.14 - jupyter-client==7.4.9 - jupyter-lsp==2.2.0 - jupyter-server-fileid==0.9.0 - jupyter-server-mathjax==0.2.6 - jupyter-server-proxy==3.2.1 - jupyter-server-ydoc==0.8.0 - jupyter-ydoc==0.2.5 - jupyterlab==3.6.5 - jupyterlab-git==0.41.0 - jupyterlab-lsp==4.2.0 - jupyterlab-server==2.24.0 - lazy-object-proxy==1.9.0 - mccabe==0.7.0 - multiprocess==0.70.15 - nbdime==3.2.1 - nbexamples==0.0.0 - pathos==0.3.1 - pid==3.0.4 - pluggy==1.2.0 - pox==0.3.3 - ppft==1.7.6.7 - py4j==0.10.9.5 - pyasn1==0.5.0 - pycodestyle==2.10.0 - pydocstyle==6.3.0 - pyflakes==3.0.1 - pygal==3.0.0 - pylint==2.17.5 - pyspark==3.3.0 - python-lsp-jsonrpc==1.0.0 - python-lsp-server==1.7.4 - pytoolconfig==1.2.5 - rope==1.9.0 - rsa==4.7.2 - sagemaker==2.179.0 - sagemaker-experiments==0.1.45 - sagemaker-nbi-agent==1.0 - sagemaker-pyspark==1.4.5 - schema==0.7.5 - simpervisor==1.0.0 - smdebug-rulesconfig==1.0.1 - smmap==5.0.0 - snowballstemmer==2.2.0 - tblib==1.7.0 - tomli==2.0.1 - tomlkit==0.12.1 - ujson==5.8.0 - urllib3==1.26.14 - y-py==0.6.0 - ypy-websocket==0.8.4 prefix: /home/ec2-user/anaconda3/envs/JupyterSystemEnv ```
pip_packages
``` accelerate==0.22.0 aiohttp==3.8.5 aiosignal==1.3.1 anyio==4.0.0 appdirs==1.4.4 asgiref==3.7.2 async-timeout==4.0.3 attrs==23.1.0 auto-gptq==0.4.2 bentoml==1.1.4 bitsandbytes==0.41.1 build==0.10.0 cattrs==23.1.2 certifi==2023.7.22 charset-normalizer==3.2.0 circus==0.18.0 click==8.1.7 click-option-group==0.5.6 cloudpickle==2.2.1 cmake==3.27.2 coloredlogs==15.0.1 contextlib2==21.6.0 cuda-python==12.2.0 Cython==3.0.2 datasets==2.14.4 deepmerge==1.1.0 Deprecated==1.2.14 dill==0.3.7 exceptiongroup==1.1.3 fastcore==1.5.29 filelock==3.12.3 filetype==1.2.0 frozenlist==1.4.0 fs==2.4.16 fsspec==2023.6.0 ghapi==1.0.4 h11==0.14.0 httpcore==0.17.3 httpx==0.24.1 huggingface-hub==0.16.4 humanfriendly==10.0 idna==3.4 importlib-metadata==6.0.1 inflection==0.5.1 Jinja2==3.1.2 lit==16.0.6 markdown-it-py==3.0.0 MarkupSafe==2.1.3 mdurl==0.1.2 mpmath==1.3.0 multidict==6.0.4 multiprocess==0.70.15 mypy-extensions==1.0.0 networkx==3.1 numpy==1.25.2 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-cupti-cu11==11.7.101 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 nvidia-cufft-cu11==10.9.0.58 nvidia-curand-cu11==10.2.10.91 nvidia-cusolver-cu11==11.4.0.1 nvidia-cusparse-cu11==11.7.4.91 nvidia-nccl-cu11==2.14.3 nvidia-nvtx-cu11==11.7.91 openllm==0.2.27 openllm-client==0.2.27 openllm-core==0.2.27 opentelemetry-api==1.18.0 opentelemetry-instrumentation==0.39b0 opentelemetry-instrumentation-aiohttp-client==0.39b0 opentelemetry-instrumentation-asgi==0.39b0 opentelemetry-sdk==1.18.0 opentelemetry-semantic-conventions==0.39b0 opentelemetry-util-http==0.39b0 optimum==1.12.0 orjson==3.9.5 packaging==23.1 pandas==2.1.0 pathspec==0.11.2 peft==0.5.0 Pillow==10.0.0 pip-requirements-parser==32.0.1 pip-tools==7.3.0 prometheus-client==0.17.1 protobuf==4.24.2 psutil==5.9.5 pyarrow==13.0.0 pydantic==1.10.12 Pygments==2.16.1 pynvml==11.5.0 pyparsing==3.1.1 pyproject_hooks==1.0.0 python-dateutil==2.8.2 python-json-logger==2.0.7 python-multipart==0.0.6 pytz==2023.3 PyYAML==6.0.1 pyzmq==25.1.1 regex==2023.8.8 requests==2.31.0 rich==13.5.2 rouge==1.0.1 safetensors==0.3.3 schema==0.7.5 scipy==1.11.2 sentencepiece==0.1.99 simple-di==0.1.5 six==1.16.0 sniffio==1.3.0 starlette==0.28.0 sympy==1.12 tabulate==0.9.0 tokenizers==0.13.3 tomli==2.0.1 torch==2.0.1 tornado==6.3.3 tqdm==4.66.1 transformers==4.32.1 triton==2.0.0 typing_extensions==4.7.1 tzdata==2023.3 urllib3==2.0.4 uvicorn==0.23.2 watchfiles==0.20.0 wcwidth==0.2.6 wrapt==1.15.0 xxhash==3.3.0 yarl==1.9.2 zipp==3.16.2 ```
transformers
version: 4.32.1System information (Optional)
No response