bug: When running by example getting error: TypeError: 'dict' object is not callable

lolevsky commented 11 months ago

Describe the bug

I am following the example and wrote the code:

llm = OpenLLM(server_url=server_url, server_type='http')
llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")

Seems like First request is hitting the server

(scheme=http,method=POST,path=/v1/metadata,type=application/json,length=2) (status=200

Till now its look promising, but then I am getting error TypeError: 'dict' object is not callable.

As attached in the trace:

Traceback (most recent call last):
  File "/home/lolevsky/Github/Zodiac/main.py", line 24, in <module>
    run_zodiac()
  File "/home/lolevsky/Github/Zodiac/main.py", line 9, in run_zodiac
    resA = llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
  File "/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py", line 876, in __call__
    self.generate(
  File "/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py", line 626, in generate
    params = self.dict()
  File "/usr/local/lib/python3.10/dist-packages/langchain/llms/base.py", line 974, in dict
    starter_dict = dict(self._identifying_params)
  File "/usr/local/lib/python3.10/dist-packages/langchain/llms/openllm.py", line 220, in _identifying_params
    self.llm_kwargs.update(self._client._config())
TypeError: 'dict' object is not callable

To reproduce

This is how I had setup the envirment:

conda create --name openllm python=3.11
conda activate openllm
pip install openllm
pip install langchain

Logs

No response

Environment

Environment variable

BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''

System information

bentoml: 1.1.10 python: 3.11.5 platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35 uid_gid: 1000:1000 conda: 23.7.4 in_conda_env: True

conda_packages

```yaml name: openllm channels: - defaults dependencies: - _libgcc_mutex=0.1=main - _openmp_mutex=5.1=1_gnu - bzip2=1.0.8=h7b6447c_0 - ca-certificates=2023.08.22=h06a4308_0 - ld_impl_linux-64=2.38=h1181459_1 - libffi=3.4.4=h6a678d5_0 - libgcc-ng=11.2.0=h1234567_1 - libgomp=11.2.0=h1234567_1 - libstdcxx-ng=11.2.0=h1234567_1 - libuuid=1.41.5=h5eee18b_0 - ncurses=6.4=h6a678d5_0 - openssl=3.0.12=h7f8727e_0 - pip=23.3.1=py311h06a4308_0 - python=3.11.5=h955ad1f_0 - readline=8.2=h5eee18b_0 - setuptools=68.0.0=py311h06a4308_0 - sqlite=3.41.2=h5eee18b_0 - tk=8.6.12=h1ccaba5_0 - wheel=0.41.2=py311h06a4308_0 - xz=5.4.2=h5eee18b_0 - zlib=1.2.13=h5eee18b_0 - pip: - accelerate==0.24.1 - aiohttp==3.9.0 - aiosignal==1.3.1 - anyio==3.7.1 - appdirs==1.4.4 - asgiref==3.7.2 - attrs==23.1.0 - beautifulsoup4==4.12.2 - bentoml==1.1.10 - bitsandbytes==0.41.2.post2 - build==0.10.0 - cattrs==23.1.2 - certifi==2023.11.17 - charset-normalizer==3.3.2 - circus==0.18.0 - click==8.1.7 - click-option-group==0.5.6 - cloudpickle==3.0.0 - coloredlogs==15.0.1 - contextlib2==21.6.0 - cuda-python==12.3.0 - dataclasses-json==0.6.2 - datasets==2.15.0 - deepmerge==1.1.0 - deprecated==1.2.14 - dill==0.3.7 - distlib==0.3.7 - distro==1.8.0 - einops==0.7.0 - fastapi==0.104.1 - fastcore==1.5.29 - filelock==3.13.1 - filetype==1.2.0 - frozenlist==1.4.0 - fs==2.4.16 - fsspec==2023.10.0 - ghapi==1.0.4 - greenlet==3.0.1 - h11==0.14.0 - httpcore==1.0.2 - httptools==0.6.1 - httpx==0.25.2 - huggingface-hub==0.19.4 - humanfriendly==10.0 - idna==3.6 - importlib-metadata==6.8.0 - inflection==0.5.1 - jinja2==3.1.2 - jsonpatch==1.33 - jsonpointer==2.4 - jsonschema==4.20.0 - jsonschema-specifications==2023.11.1 - langchain==0.0.340 - langsmith==0.0.66 - markdown-it-py==3.0.0 - markupsafe==2.1.3 - marshmallow==3.20.1 - mdurl==0.1.2 - mpmath==1.3.0 - msgpack==1.0.7 - multidict==6.0.4 - multiprocess==0.70.15 - mypy-extensions==1.0.0 - networkx==3.2.1 - ninja==1.11.1.1 - numpy==1.26.2 - nvidia-cublas-cu12==12.1.3.1 - nvidia-cuda-cupti-cu12==12.1.105 - nvidia-cuda-nvrtc-cu12==12.1.105 - nvidia-cuda-runtime-cu12==12.1.105 - nvidia-cudnn-cu12==8.9.2.26 - nvidia-cufft-cu12==11.0.2.54 - nvidia-curand-cu12==10.3.2.106 - nvidia-cusolver-cu12==11.4.5.107 - nvidia-cusparse-cu12==12.1.0.106 - nvidia-ml-py==11.525.150 - nvidia-nccl-cu12==2.18.1 - nvidia-nvjitlink-cu12==12.3.101 - nvidia-nvtx-cu12==12.1.105 - openllm==0.4.28 - openllm-client==0.4.28 - openllm-core==0.4.28 - opentelemetry-api==1.20.0 - opentelemetry-instrumentation==0.41b0 - opentelemetry-instrumentation-aiohttp-client==0.41b0 - opentelemetry-instrumentation-asgi==0.41b0 - opentelemetry-sdk==1.20.0 - opentelemetry-semantic-conventions==0.41b0 - opentelemetry-util-http==0.41b0 - optimum==1.14.1 - orjson==3.9.10 - packaging==23.2 - pandas==2.1.3 - pathspec==0.11.2 - pillow==10.1.0 - pip-requirements-parser==32.0.1 - pip-tools==7.3.0 - platformdirs==4.0.0 - prometheus-client==0.19.0 - protobuf==4.25.1 - psutil==5.9.6 - pyarrow==14.0.1 - pyarrow-hotfix==0.6 - pydantic==1.10.13 - pygments==2.17.2 - pyparsing==3.1.1 - pyproject-hooks==1.0.0 - python-dateutil==2.8.2 - python-dotenv==1.0.0 - python-json-logger==2.0.7 - python-multipart==0.0.6 - pytz==2023.3.post1 - pyyaml==6.0.1 - pyzmq==25.1.1 - ray==2.8.0 - referencing==0.31.0 - regex==2023.10.3 - requests==2.31.0 - rich==13.7.0 - rpds-py==0.13.1 - safetensors==0.4.0 - schema==0.7.5 - scipy==1.11.4 - sentencepiece==0.1.99 - simple-di==0.1.5 - six==1.16.0 - sniffio==1.3.0 - soupsieve==2.5 - sqlalchemy==2.0.23 - starlette==0.27.0 - sympy==1.12 - tenacity==8.2.3 - tokenizers==0.15.0 - torch==2.1.0 - tornado==6.3.3 - tqdm==4.66.1 - transformers==4.35.2 - triton==2.1.0 - typing-extensions==4.8.0 - typing-inspect==0.9.0 - tzdata==2023.3 - urllib3==2.1.0 - uvicorn==0.24.0.post1 - uvloop==0.19.0 - virtualenv==20.24.7 - vllm==0.2.2 - watchfiles==0.21.0 - websockets==12.0 - wrapt==1.16.0 - xformers==0.0.22.post7 - xxhash==3.4.1 - yarl==1.9.3 - zipp==3.17.0 prefix: /home/lolevsky/anaconda3/envs/openllm ```

pip_packages

``` accelerate==0.24.1 aiohttp==3.9.0 aiosignal==1.3.1 anyio==3.7.1 appdirs==1.4.4 asgiref==3.7.2 attrs==23.1.0 beautifulsoup4==4.12.2 bentoml==1.1.10 bitsandbytes==0.41.2.post2 build==0.10.0 cattrs==23.1.2 certifi==2023.11.17 charset-normalizer==3.3.2 circus==0.18.0 click==8.1.7 click-option-group==0.5.6 cloudpickle==3.0.0 coloredlogs==15.0.1 contextlib2==21.6.0 cuda-python==12.3.0 dataclasses-json==0.6.2 datasets==2.15.0 deepmerge==1.1.0 Deprecated==1.2.14 dill==0.3.7 distlib==0.3.7 distro==1.8.0 einops==0.7.0 fastapi==0.104.1 fastcore==1.5.29 filelock==3.13.1 filetype==1.2.0 frozenlist==1.4.0 fs==2.4.16 fsspec==2023.10.0 ghapi==1.0.4 greenlet==3.0.1 h11==0.14.0 httpcore==1.0.2 httptools==0.6.1 httpx==0.25.2 huggingface-hub==0.19.4 humanfriendly==10.0 idna==3.6 importlib-metadata==6.8.0 inflection==0.5.1 Jinja2==3.1.2 jsonpatch==1.33 jsonpointer==2.4 jsonschema==4.20.0 jsonschema-specifications==2023.11.1 langchain==0.0.340 langsmith==0.0.66 markdown-it-py==3.0.0 MarkupSafe==2.1.3 marshmallow==3.20.1 mdurl==0.1.2 mpmath==1.3.0 msgpack==1.0.7 multidict==6.0.4 multiprocess==0.70.15 mypy-extensions==1.0.0 networkx==3.2.1 ninja==1.11.1.1 numpy==1.26.2 nvidia-cublas-cu12==12.1.3.1 nvidia-cuda-cupti-cu12==12.1.105 nvidia-cuda-nvrtc-cu12==12.1.105 nvidia-cuda-runtime-cu12==12.1.105 nvidia-cudnn-cu12==8.9.2.26 nvidia-cufft-cu12==11.0.2.54 nvidia-curand-cu12==10.3.2.106 nvidia-cusolver-cu12==11.4.5.107 nvidia-cusparse-cu12==12.1.0.106 nvidia-ml-py==11.525.150 nvidia-nccl-cu12==2.18.1 nvidia-nvjitlink-cu12==12.3.101 nvidia-nvtx-cu12==12.1.105 openllm==0.4.28 openllm-client==0.4.28 openllm-core==0.4.28 opentelemetry-api==1.20.0 opentelemetry-instrumentation==0.41b0 opentelemetry-instrumentation-aiohttp-client==0.41b0 opentelemetry-instrumentation-asgi==0.41b0 opentelemetry-sdk==1.20.0 opentelemetry-semantic-conventions==0.41b0 opentelemetry-util-http==0.41b0 optimum==1.14.1 orjson==3.9.10 packaging==23.2 pandas==2.1.3 pathspec==0.11.2 Pillow==10.1.0 pip-requirements-parser==32.0.1 pip-tools==7.3.0 platformdirs==4.0.0 prometheus-client==0.19.0 protobuf==4.25.1 psutil==5.9.6 pyarrow==14.0.1 pyarrow-hotfix==0.6 pydantic==1.10.13 Pygments==2.17.2 pyparsing==3.1.1 pyproject_hooks==1.0.0 python-dateutil==2.8.2 python-dotenv==1.0.0 python-json-logger==2.0.7 python-multipart==0.0.6 pytz==2023.3.post1 PyYAML==6.0.1 pyzmq==25.1.1 ray==2.8.0 referencing==0.31.0 regex==2023.10.3 requests==2.31.0 rich==13.7.0 rpds-py==0.13.1 safetensors==0.4.0 schema==0.7.5 scipy==1.11.4 sentencepiece==0.1.99 simple-di==0.1.5 six==1.16.0 sniffio==1.3.0 soupsieve==2.5 SQLAlchemy==2.0.23 starlette==0.27.0 sympy==1.12 tenacity==8.2.3 tokenizers==0.15.0 torch==2.1.0 tornado==6.3.3 tqdm==4.66.1 transformers==4.35.2 triton==2.1.0 typing-inspect==0.9.0 typing_extensions==4.8.0 tzdata==2023.3 urllib3==2.1.0 uvicorn==0.24.0.post1 uvloop==0.19.0 virtualenv==20.24.7 vllm==0.2.2 watchfiles==0.21.0 websockets==12.0 wrapt==1.16.0 xformers==0.0.22.post7 xxhash==3.4.1 yarl==1.9.3 zipp==3.17.0 ```

System information (Optional)

No response

aarnphm commented 11 months ago

There is an upstream langchain PR yet to be merged. https://github.com/langchain-ai/langchain/pull/12968.

aarnphm commented 11 months ago

Once that one is merged you can install the patch with pip install "git+https://github.com/bentoml/langchain.git@chore/migrate-to-new-api#subdirectory=libs/langchain"

newparad1gm commented 11 months ago

I tried that update, but I was still getting errors in openllm.py at lines 226-227 that the Metadata object was not subscriptable. I had to change code to this

            model_name = self._client._metadata.model_name
            model_id = self._client._metadata.model_id

MexHigh commented 10 months ago

I also have the exact same problem. Even downgrading langchain/openllm for several minors and majors didn't help. I'm in a fresh anaconda environment.

Once that one is merged you can install the patch with pip install "git+https://github.com/bentoml/langchain.git@chore/migrate-to-new-api#subdirectory=libs/langchain"

@aarnphm Is there a way to get this working before langchain merges this PR? Which version should I use then?

aarnphm commented 10 months ago

YOu can install langchain from the pip command.

aarnphm commented 10 months ago

What is the issue you are running into?

bojiang commented 3 months ago

close for openllm 0.6

bentoml / OpenLLM